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ABSTRACT 


Test  and  evaluation  of  system  performance  has  been  a  critical  part  of  the 
acceptance  of  combat  weapon  systems  for  the  Department  of  Defense.  As 
combat  weapon  systems  have  become  more  complex,  evaluation  of  system 
performance  has  relied  more  heavily  on  recorded  test  data.  As  part  of  the  on¬ 
going  transformation  of  the  Defense  department,  Commercial-Off-The-Shelf 
(COTS)  technology  is  being  integrated  into  the  acquisition  of  combat  weapon 
systems.  An  Analysis  Control  Board  (ACB)  was  created  in  response  to  these 
factors  to  support  the  AEGIS  Weapon  System  Program  Office.  The  focus  of  this 
ACB  was  to  investigate  and  provide  potential  solutions  to  Data  Dictionary,  Data 
Recording  and  Data  Reduction  (R2D2)  issues  to  the  AEGIS  Program  Manager. 
This  thesis  discusses  the  history  of  the  R2D2  ACB  and  its  past,  present  and 
future  directions.  Additionally,  this  thesis  examines  how  the  R2D2  ACB  concept 
could  be  applied  to  the  DD(X)  Next  Generation  Destroyer  program. 
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EXECUTIVE  SUMMARY 


Test  and  Evaluation  is  a  critical  part  of  the  procurement  of  military  systems 
and  analysis  of  test  data  is  required  to  determine  if  the  system  performance  met 
the  defined  requirements.  In  an  effort  to  ensure  that  the  test  data  from  formal 
acceptance  tests,  such  as  Ship  Qualification  Trials,  Developmental  and 
Operation  Tests  was  adequate  for  analysis,  an  analysis  control  board  (ACB)  was 
formed  to  address  Data  Dictionary,  Recording  and  Reduction  (R2D2)  issues. 
The  R2D2  ACB  has  supported  program  managers  in  the  AEGIS  program  office 
by  investigating  and  providing  technical  expertise  for  Data  Dictionary,  Recording 
and  Reduction  issues.  The  R2D2  ACB  is  a  proactive  effort  sponsored  by  the 
AEGIS  program  office  to  resolve  issues  that  will  affect  the  evaluation  of  formal 
acceptance  testing  before  the  tests  are  executed.  Resolving  issues  prior  to 
formal  acceptance  testing  significantly  reduces  the  risk  of  requiring  a  test  to  be 
re-executed.  As  the  Defense  Department  transforms  itself  to  meet  future 
missions,  this  study  examines  Defense  Transformation  and  SEA  POWER  21  to 
determine  how  the  R2D2  ACB  can  change  to  meet  the  challenges  of  the  future. 

Changes  in  the  design  of  military  systems,  specifically  the  implementation 
of  Commercial-Off-The-Shelf  hardware,  have  provided  significant  challenges  to 
the  recording  and  analysis  of  test  data  from  the  AEGIS  Weapon  System.  As  the 
AEGIS  Weapon  System  completes  the  transition  to  Open  Architecture,  the 
challenge  of  assessing  system  performance  will  become  even  greater.  This 
study  includes  an  analysis  of  the  DD(X)  Next  Generation  Destroyer  program  and 
how  the  concept  of  the  R2D2  ACB  can  be  applied  to  this  program. 
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I.  INTRODUCTION 


A.  BACKGROUND 

During  the  early  development  of  the  AEGIS  computer  program,  data  was 
recorded  in  a  defined  format  called  AEGIS  Tactical  Executive  System  for  the 
AN/UYK-43  (ATES/43)1,  or  more  commonly  referred  to  as  ATES.  The  function  of 
data  recording  is  to  provide  an  objective  set  of  information  to  evaluate  system 
performance.  Historically,  the  AEGIS  program  has  recorded  data  in  a  binary 
format  with  the  ATES  format  providing  a  “data  recording  function,  which  provides 
the  means  for  extracting  selected  main  memory  resident  data  from  tactical 
computer  programs  and  for  recording  this  data  on  media  (optical  disk  or 
magnetic  tape)  for  subsequent  reduction  and  analysis.”  (“Program  Performance 
Specification”,  1999,  p.  1-1)  ATES  was  customized  to  allow  the  data  to  be 
captured  from  the  unique,  military  standard  hardware.  This  recording  method 
focused  on  dumping  data  that  was  recorded  in  specific,  defined,  hard  coded 
memory  locations.  Since  the  hardware  was  military  standard  (MIL-STD)  and 
non-commercial,  this  method  proved  effective  within  the  constraints  of  non¬ 
commercial  hardware  and  software.  The  mission  of  ATES  is  defined  “...  as  a 
subsystem  within  each  of  the  combat  system  computer  programs  listed  ...  to 
provide  for  each  of  them  an  environment  within  which  they  can  achieve  their 
individual  missions,  and  can  work  together  to  achieve  the  mission  of  the  Combat 
System.  This  environment  provides  the  ability  for  a  program  module  to 
communicate  with  other  program  modules  and  data  internal  to  the  AN/UYK-43, 
the  AN/UYK-43  hardware,  and,  via  this  hardware,  the  devices  outside  the 
AN/UYK-43.”  (“Program  Performance  Specification”,  1999,  p.  1-1)  The  data 
recording  function  was  designed  into  the  system  during  development  providing 
an  inherent  advantage  to  later  systems  for  event  reconstruction  but  it  was  tied  to 

1  The  ATES  data-recording  format  was  designed  to  be  used  with  MIL-SPEC  computers  by 
the  AEGIS  program.  DXR  was  introduced  with  the  insertion  of  COTS  computers  into  the  AEGIS 
Weapon  System.  ATES  and  DXR  format  the  data  received  from  the  AWS  and  write  the  data  in  a 
binary  format  to  a  storage  media,  generally  a  digital  tape  or  optical  disk.  When  this  data  is 
extracted  from  the  storage  media,  a  data  dictionary  is  used  to  reconstruct  the  data  into  the 
appropriate  fields  of  information. 
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customized,  MIL-STD  hardware.  ATES  had  some  very  specific  capabilities 
embedded  into  the  design  of  the  data  recording  function  including  controlling 
application  program  module  execution,  fault  tolerance  to  recover  from  all  single 
point  failures,  support  inter-computer  communications  and  the  data  environment 
including  data  recording.  (“Program  Performance  Specification”,  1999,  p.  1-1) 

In  the  early  development  of  the  AEGIS  Weapon  System,  data  recording 
and  extraction  was  a  critical  component  of  the  development.  The  system 
designers  and  integrators  relied  upon  the  data  extraction  to  determine  system 
performance  and  specification  compliance.  However,  data  recording  is  not 
unique  to  the  AEGIS  program.  Data  recording  and  extraction  is  critical  to  test 
and  evaluation  events  in  order  to  determine  whether  the  system  performance 
was  acceptable.  Whether  the  data  is  recorded  on  a  local  disk  drive  or  captured 
by  telemetry  from  a  satellite,  the  ability  to  reconstruct  system  performance  is 
critical  to  determining  is  the  mission  was  successfully  accomplished. 

Baseline  migration  to  Commercial-Off-the-Shelf  (COTS)  technology 
presented  numerous  data  extraction/data  reduction  (DX/DR)  challenges.  A 
meeting  was  held  at  the  Combat  Systems  Engineering  Development  Site 
(CSEDS),  Moorestown,  NJ  on  25  July  2001  to  discuss  DX/DR  issues  associated 
with  AEGIS  Display  System  (ADS)  Mark  6  and  AEGIS  Weapon  System  (AWS) 
Baseline  6  Phase  3.  The  discussion  was  focused  on  identifying  current  data 
extraction  and  reduction  (DX/DR)  short  falls,  assessing  their  impact  on  upcoming 
test  events,  and  ensuring  a  solid  plan  was  in  place  to  correct  discrepancies.  Two 
major  areas  of  ADS  Mark  6  concerns  were  presented, 

1 .  Inability  to  perform  complete  event  reconstruction  at  ADS  and 

2.  Data  dictionary  deficiencies  limit  the  ability  to  reconstruct  a 
complete  set  of  console  operator  actions,  operator  alerts  and 
doctrine. 

Cooperative  Engagement  Capability  (CEC)  provided  the  first  at-sea 
developmental  testing  for  the  ADS  Mark  6  aboard  the  USS  HUE  CITY  and  USS 
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VICKSBURG.  The  at-sea  testing  resulted  in  a  large  number  of  high  priority 
issues.  The  lack  of  DX/DR  capability  for  ADS  Mark  6  was  overshadowed  as  a 
result  of  the  multitude  of  problems.  The  limited  time  and  resources  that  were 
available  at  this  time  were  devoted  to  developing  the  tactical  baseline  code  that 
was  viewed  as  the  highest  priority.  As  such,  data  analysis  capability  in  ADS 
Mark  6  received  a  low  priority  during  the  two  years  of  shipboard  testing.  While 
an  ADS  Baseline  6  Phase  1R  data  dictionary  was  provided  during  the  later 
phases  of  Cooperative  Engagement  Capability  (CEC)  testing,  enumerations  to 
interpret  numeric  data  were  not  provided,  precluding  meaningful  analysis  from 
being  accomplished.  Since  there  are  no  Extraction  Points  (EP)  in  ADS  data 
dictionary  for  the  track  database  corresponding  the  data  displayed  at  the  ADS 
consoles,  complete  reconstruction  of  tactical  events  as  they  were  processed  by 
key  watch  standers  (e.g.  Commanding  Officer  (CO),  Tactical  Action  Officer 
(TAO))  at  all  Q-70  consoles  was  not  possible.  The  lack  of  complete 
reconstruction  of  the  tactical  picture  was  a  departure  from  the  previous  ATES 
baselines.  The  impact  of  this  condition  varies  in  severity  according  to  the 
implementation  of  ADS  in  each  particular  baseline.  Recent  AEGIS  baselines 
have  captured  some  of  this  lost  capability. 

Since  the  implementation  of  COTS  technology  for  AWS  was  a  phased 
approach,  the  analysis  community  realized  that  the  problems  faced  with  ADS 
were  a  precursor  to  similar  problems  that  could  result  in  the  COTS 
implementation  of  the  other  elements  of  AWS.  The  length  of  production  runs  for 
COTS  components  is  significantly  shorter  than  the  comparable  MIL-SPEC 
components  historically  used  in  the  AWS.  The  result  of  this  was  differing 
performance  among  equipment  within  the  same  AEGIS  baseline.  Additionally, 
the  differences  in  COTS  implementation  between  Cruisers  and  Destroyers 
resulted  in  multiple  variants  of  the  same  AEGIS  baseline.  Different 
configurations  within  the  Destroyers  resulted  in  variants  within  the  AEGIS 
Destroyers  for  Baseline  6  Phase  1 .  The  spawning  of  multiple  variants  of  the 
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AEGIS  baselines  resulted  in  large  efforts  to  make  the  tactical  computer  programs 
ready  for  test  and  resulted  in  a  resource  limitation  for  non-tactical  areas  including 
data  recording  and  extraction. 

While  the  problems  with  the  ADS  data  recording  provided  the  groundswell 
of  concern  that  launched  the  Data  Dictionary,  Recording  and  Reduction  Analysis 
Control  Board  (R2D2  ACB),  the  R2D2  ACB  evaluates  issues  with  recorded  data 
for  the  SPY-1  radar,  command  and  decision  (C&D)  and  weapon  control  systems 
(WCS)  as  well.  An  analysis  control  board  (ACB)  is  a  technical  working  group  that 
is  chartered  to  deal  specifically  with  data  analysis  issues.  The  ACB  meetings  are 
scheduled  based  upon  the  testing  schedule  and  number  and  priority  of 
outstanding  issues.  During  the  year  2003,  the  R2D2  ACB  met  approximately 
once  every  two  months.  Participants  in  the  ACB  included  Naval  Sea  Systems 
Command,  Naval  Surface  Warfare  Centers  (Corona,  Dahlgren  and  Port 
Hueneme  Divisions),  Lockheed  Martin,  Northrop  Grumman  Mission  Systems  and 
others. 

The  R2D2  ACB  is  part  of  the  closed  loop  system  engineering  process  that 
is  used  by  the  AEGIS  program.  Within  this  format,  the  R2D2  ACB  reports  to  the 
program  manager  and  is  available  to  support  the  system  engineering  councils  as 
required.  The  Closed  Loop  System  Engineering  process  as  implemented  in  the 
AEGIS  Program  is  illustrated  in  Figure  1. 
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•  System  Engineering  Requirements 
•  Operational  Requirements 

•  Previous  Deficiencies 
•  Added  Capabilities 

•  Previous  Test  Results 

•  Simulation/ Validation 


•  Modify  Requirements 
•  Correct  Training  Deficiencies 

•  Correct  Performance  Deficiencies 

•  Improve  Capability 

•  Determine  Performance  Umiters 

•  Improve  M&S  Predictions 
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•  Test  Objectives  (TOs) 

•  Measures  of  Effectiveness 
•  Test  Kinematics 


•  TO  Assignment  to  Test  Event 

4 

•  Scenario  Development 
•  Performance  Predictions 
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•  Test  Plan 
•  Analysis  Plan 


Plan  Certification 


t 

•  Performance  Assessment 
•  Test  Objective  Assessment 
•  Daily  Reports 


•  Observations 
■  Results 


i 

•  Test  Execution 


Figure  1.  Closed  Loop  System  Engineering  Process. 

(Sparks,  2001) 


In  this  closed  loop  system  engineering  (CLSE)  process,  system 
engineering  requirements,  operational  requirements,  deficiencies,  new  added 
capabilities,  previous  test  results,  and  simulation/validation  issues  are  evaluated. 
Test  objectives  are  created  to  ensure  that  the  following  areas  are  properly 
analyzed.  Measures  of  Effectiveness  specify  how  each  test  objective  is 
assessed  and  includes  test  kinematics,  such  as  the  required  sequence  of  events, 
system  settings,  and  target  needs.  Test  objectives  are  assigned  to  specific  test 
events.  Test  scenarios  are  developed  to  meet  the  objectives.  Modeling  and 
Simulation  studies  are  performed  to  predict  the  expected  outcome  of  each  test 
scenario.  Next,  the  requirements  to  execute  the  test  scenarios  are  documented 
in  the  test  plan.  The  test  plan  is  required  to  complete  a  certification  process  that 
is  completed  about  three  months  prior  to  the  beginning  of  testing  to  allow 
sufficient  time  for  test  scenario  preparations.  During  the  CSSQT  testing,  the  test 
scenarios  are  executed  and  the  test  data  is  collected  and  delivered  to  the 
analysis.  Test  objectives  are  assessed  and  performance  issues  become  part  of 
the  Weapon  System  Performance  Assessment  (WSPA)  process.  Together,  test 

objective  assessment  and  the  WSPA  process  feedback  into  the  process, 
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resulting  in  modification  of  requirements,  correction  of  training  and  performance 
deficiencies,  capabilities  improvements,  determination  of  performance  limiters, 
and  improvement  of  M&S  models.  The  test  results  are  then  fed  into  new  test 
objectives  and  the  closed  loop  system  engineering  process  continues  on.  The 
R2D2  ACB  is  part  of  the  assessment,  reporting  and  system  engineering 
requirements  functions  of  this  closed  loop  system  engineering  process. 

B.  PURPOSE  AND  RESEARCH  QUESTIONS 

The  purpose  of  this  study  is  to  explore  the  lessons  learned  from  the  R2D2 
ACB  and  the  value,  if  any,  that  it  provides.  This  study  will  investigate  the 
potential  value  that  this  type  of  forum  can  provided  to  other  military  weapon 
systems.  The  following  research  questions  have  been  formulated  to  explore 
R2D2  ACB  efforts  and  potential  future  directions. 


•  Has  the  R2D2  ACB  provided  value  to  the  AEGIS  program? 

•  What  changes,  if  any,  can  be  incorporated  into  the  R2D2  ACB  using  best 
practices  available  today? 

•  Is  the  Navy  able  to  spend  sufficient  time  supporting  R2D2  issues? 

•  What  level  of  effort  is  required  to  successfully  institute  an  R2D2  ACB  and 

what  level  of  support  does  the  management/leadership  need  to  provide  for 
the  ACB  to  provide  value  to  each  participant’s  activity? 

•  Can  the  R2D2  ACB  concept  be  applied  to  the  DDX  program? 

•  Can  the  R2D2  concept  be  applied  to  AEGIS  Open  Architecture? 

C.  STAKEHOLDER  INFORMATION 


There  are  many  stakeholders  in  the  R2D2  ACB.  Each  stakeholder  is 
listed  below  with  a  brief  description: 
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•  Naval  Sea  Systems  Command  (NAVSEA)  Program  Executive  Office 
Integrated  Warfare  Systems  (PEO-IWS):  Program  office  responsible  for 
assignment  of  tasking.  PEO-IWS  1A1B  has  oversight  of  the  R2D2  ACB  for 
the  AEGIS  program. 

•  Naval  Surface  Warfare  Center  (NSWC),  Corona  Division:  Tasked  to  be 
the  independent  assessment  agent  for  NAVSEA/PEO-IWS  1  A. 

•  Naval  Surface  Warfare  Center,  Dahlgren  Division  (NSWC  DD): 

Responsible  for  AEGIS  computer  program  certification. 

•  Naval  Surface  Warfare  Center,  Port  Hueneme  Division:  Performs  the  In- 
Service  Engineering  Agent  (ISEA)  function. 

•  Computer  Services  Corporation  (CSC):  Responsible  for  developing 
computer  code  for  the  AEGIS  program. 

•  Lockheed  Martin  Maritime  and  Marine  Systems  (LM  MMS):  The  prime 
contractor  and  design  agent  for  the  AEGIS  program. 

•  Combat  Systems  Engineering  Development  Site  (CSEDS):  Government 
owned  facility  where  AEGIS  computer  program  baselines  are  tested  and 
functionality  is  demonstrated. 

•  Northrop  Grumman  Mission  Systems  (NGMS):  Contractor  that  supports 
the  AEGIS  program.  NGMS  has  offices  in  Washington,  DC  to  support  the 
Program  Office  at  WNY  and  in  Mt.  Laurel,  NJ  to  support  CSEDS. 


D.  SURVEY  QUESTIONS 


It  is  important  to  ensure  that  a  representative  of  each  stakeholder  respond 
to  the  survey  to  ensure  that  the  results  reflect  a  consensus  view.  There  are 
many  methods  that  can  be  used  and  will  be  discussed  in  section  3.  The  survey 
is  designed  to  provide  insight  into  answering  the  research  questions. 


E.  SCOPE 


The  scope  of  this  study  will  examine  three  specific  areas:  AEGIS  Baseline 
7  Phase  1,  DD(X)  platform  and  AEGIS  Open  Architecture  (AOA).  While  the 
DD(X)  platform  is  programmed  to  be  an  Open  Architecture  platform,  it  is  unclear 


7 


as  to  the  level  of  compatibility  that  the  DD(X)  and  AOA  implementations  of  Open 
Architecture  will  have. 

The  R2D2  ACB  has  come  into  existence  out  of  necessity.  However,  since 
its  inception,  no  effort  has  been  undertaken  to  quantify  the  benefits  that  it 
provides.  This  study  will  attempt  to  quantify  the  benefits  that  the  R2D2  ACB 
provides  to  the  AEGIS  program.  A  high  level  review  of  efforts  involved  in 
supporting  the  ACB  will  also  be  characterized  and  compared  to  those  benefits. 
After  this  comparison  is  completed,  the  DD(X)  and  Open  Architecture  programs 
will  be  examined  to  determine  if  the  benefits  provided  to  the  AEGIS  program  will 
continue  to  be  benefits  to  these  emerging  programs.  A  potential  scope  of  costs 
will  also  be  examined  for  comparison  to  the  expected  benefits.  This  study  will 
also  provide  a  roadmap  for  support  of  future  AEGIS  baselines  that  are  all 
programmed  to  have  some  level  of  AOA  implementation. 


F.  BENEFITS  OF  STUDY 


There  are  three  readily  identifiable  benefits  of  this  study.  The  first  benefit 
is  that  the  R2D2  ACB  will  have  a  roadmap  for  supporting  future  programs.  The 
second  benefit  is  that  the  R2D2  ACB  will  be  able  to  evaluate  its  present  practices 
to  determine  if  improvements  can  be  put  in  place.  The  third  and  most  important 
benefit  is  that  this  study  will  validate  the  value  provided  by  the  R2D2  ACB  or  if 
value  is  not  provided,  a  recommendation  to  cease  operations  will  be  made. 
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II.  LITERATURE  REVIEW 


For  this  research  study,  a  number  of  documents  have  been  reviewed. 
Primary  sources  of  information  included  R2D2  ACB  presentations,  meeting 
minutes  and  email  messages.  Additional  sources  included  specifications,  articles 
and  selected  publications. 


A.  PROGRAM  REVIEW 

1.  R2D2ACB 


Figure  2.  R2D2  ACB  Organizational  Chart. 

(Gallagher,  2004). 

An  ACB  is  a  technical  working  group  that  relies  upon  participation  from  the 
entire  AEGIS  data  analysis  community,  which  ultimately  answers  to  the  AEGIS 
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program  Office.  The  ACB  strives  to  obtain  group  consensus  for  issues  that  must 
be  resolved.  While  group  consensus  is  not  always  possible,  most  issues  are 
resolved  with  group  consensus.  In  cases  where  a  group  consensus  is  not 
reached,  a  majority  and  minority  viewpoint  are  presented  to  the  program 
manager  for  ultimate  adjudication  if  necessary.  The  R2D2  ACB  has  experienced 
development  stages  experienced  by  most  working  groups  and  passed  through 
the  stages  of  Forming,  Storming,  Norming,  and  Performing  as  depicted  below. 
One  of  the  challenges  of  this  ACB  has  been  to  avoid  “paralysis  by  analysis”  as 
the  group  is  composed  of  analysts.  By  encouraging  discussion  and  respect  of  all 
viewpoints,  the  ACB  has  been  able  to  extend  the  Performing  stage  of  its 
development.  (DAU  Program  Managers  Tool  Kit,  2003) 


WORKING  GROUPS 


TEAM  DEVELOPMENT  WHEEL 


RECOGNIZE  WHICH  PHASE  OF 
TEAM  DEVELOPMENT  YOU  ARE 
IN  AND  TAKE  POSITIVE  ACTION 
TO  WORK  THROUGH 


Note:  These  can  be  an  additional  phase  -  “Adjourning11  -  when  the  team 
disbands,  says  good  bye.  and  reflects  on  lessons  learned. 

This  is  a  ,:celebration':  phase. 


Figure  3.  Working  Group  Model. 

(DAU  Program  Managers  Tool  Kit,  2003,  p.79) 
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2.  Coordinating  Multi-agency  Testing  and  Satisfying  Individual 
Organizational  Requirements 

In  the  development  of  the  AEGIS  weapon  system,  each  computer 
program  baseline  requires  a  series  of  tests.  The  tests  required  could  include: 
functional  configuration,  demonstration,  multi-element  integration,  computer 
certification,  combat  ship  qualification  trial,  development  and  operational  testing. 
Each  of  these  tests  may  have  a  subset  of  stakeholders  within  the  ACB  and  occur 
at  different  stages  in  the  computer  program  development.  One  benefit  of 
participation  in  the  ACB  has  been  the  opportunity  for  involvement  of  all  of  the 
participants  in  earlier  stages  of  computer  program  development.  The  charter  of 
the  ACB  allows  for  a  feedback  mechanism  to  the  computer  program  developer 
and  integrator  while  keeping  the  involvement  focused  on  issues  that  provide  the 
greatest  value.  Early  on,  the  prime  contractor  (LM)  realized  that  receiving 
feedback  on  data  recording  issues  earlier  in  the  computer  program  development 
would  help  problems  to  be  fixed  before  formal  test  and  evaluation  events.  While 
having  optimal  data  recording  is  desired,  a  dynamic  balance  exists  that  requires 
consideration  of  development  of  tactical  functionality  versus  analysis  capability. 
Capturing  the  value  provided  by  the  improvements  in  analysis  capability  allows 
the  program  manager  to  determine  whether  the  benefit  provided  is  worth  the 
required  resource  expenditure. 

3.  Selection  of  Team  Members 

Analysis  Team  Member  selection  is  effectively  controlled  by  the  program 
manager  and  exercises  control  by  the  budgeting  of  program  funds.  The 
chairman  of  the  ACB  has  considerable  influence  in  the  selection  and  retention  of 
team  members  as  well.  Each  organization  that  is  involved  in  the  ACB  relies  upon 
the  program  manager  to  provide  funding  for  many  tasks  beyond  the  ACB.  The 
funding  control  allows  the  program  manager  to  determine  the  level  of 

participation  required  from  each  organization.  If  the  participation  is  insufficient, 
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the  program  manager  can  require  an  activity  to  provide  additional  members  to 
support  the  R2D2  ACB.  Similarly,  if  a  participant  is  not  supporting  the  efforts  of 
the  ACB,  the  program  manager  can  have  that  member  removed  from 
membership  in  the  R2D2  ACB. 

4.  Program  Manager  Support 

For  the  ACB  to  be  successful,  the  program  manager  must  support  the 
ACB.  Reasons  for  this  include  funding  and  acceptance  of  ACB  findings.  Without 
program  manager  support,  the  ACB  would  not  be  able  to  get  an  adequate  level 
of  participation  to  uncover  and  evaluate  data  dictionary,  data  recording  and 
reduction  issues.  Even  if  the  issues  are  discovered  and  documented,  the 
program  manager  needs  to  understand  and  prioritize  resources  to  address  each 
issue  relative  to  its  risk  to  the  program. 

5.  Communication  Methods 

Communication  methods  vary  from  informal  to  very  formal.  ACB  meetings 
are  generally  communicated  by  a  generic  email  sent  to  a  large  number  of 
participants  and  interested  parties.  More  formal  communication  has  occurred  at 
AEGIS  Weapon  System  Performance  Reviews  (WSPR)  where  the  R2D2  ACB 
presents  major  findings  and  issues  to  the  AEGIS  community.  A  website  is  used 
to  post  information  and  meeting  minutes  regarding  the  ACB  to  allow  participant  to 
review  past  events. 

6.  Entrance  Criteria 

The  entrance  criteria  for  consideration  by  the  R2D2  ACB  is  that  a 

computer  program  baseline  is  under  development  and  is  entering  or  in  the  Test 

and  Evaluation  phase.  The  exit  criterion  is  correspondingly,  when  the  Test  and 
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Evaluation  phase  is  complete  including  operational  testing  and  the  required 
analysis  reports  have  been  submitted. 


B.  DETERMINATION  OF  STAKEHOLDERS 


The  program  manager  and  the  ACB  chairman  determine  stakeholders. 
Any  organization  or  person  desiring  to  join  the  ACB  can  make  a  request  to  the 
program  manager  or  ACB  chair.  All  direct  participants  in  the  ACB  are  considered 
to  be  direct  or  indirect  stakeholders  by  virtue  of  their  ACB  status. 


C.  ROLES  AND  RESPONSIBILITIES 


The  roles  of  the  ACB  fall  into  a  few  categories  as  listed  below: 


•  Program  Manager:  Final  authority  on  whether  funding  can  be  provided  to 
remedy  data  dictionary,  recording  and  reduction  issues.  Provides  high- 
level  interface  to  the  program  manager  and  executive  management  levels 
at  the  contractor  and  within  the  Program  Executive  Office. 

•  ACB  Chairman:  Responsible  for  determining  issues  that  merit  discussion 
and  setting  agendas.  Reports  directly  to  the  program  manager  and 
provides  a  single  voice  representing  the  analysis  community  consensus 
on  issues  discussed.  The  chairman  provides  an  objective  view  that  does 
not  favor  any  individual  stakeholder  or  member. 

•  Leader:  Each  stakeholder  activity  designates  a  leader  for  their 
representation.  The  leader  is  expected  to  provide  a  consensus  view  for 
each  issue  for  their  activity.  The  leader  is  also  responsible  for  ensuring 
that  action  items  assigned  to  members  from  their  activity  are  answered. 

•  Participant:  A  participant  is  someone  who  regularly  or  periodically 
attends  ACB  meetings.  Their  level  of  involvement  varies  from  significant 
to  spectator  depending  on  issues  that  require  discussion  and  action. 
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D. 


CURRENT  EVALUATION  PLAN 


The  AEGIS  baseline  managers  evaluate  the  efforts  of  the  R2D2  ACB  and 
their  support  for  ACB  efforts  is  required  for  success.  If  the  R2D2  ACB  is  not 
performing  well,  the  baseline  manager  will  not  support  the  R2D2  ACB.  Lack  of 
participation  by  the  baseline  manager  indicates  to  the  contractor  that  the  issues 
that  the  ACB  surfaces  are  not  important  enough  to  warrant  expenditure  of 
significant  resources.  However,  if  the  baseline  manager  is  supportive  and 
involved,  the  effort  expended  to  remedy  issues  will  increase  as  the  contractor 
views  the  ACB  as  an  extension  of  the  program  office. 

E.  IDENTIFIED  RISKS  /  CONCERNS 

1.  Present  Current  Data  Dictionary,  Recording  and  Reduction 
Issues 

The  R2D2  ACB  meets  approximately  once  every  two  months  to  discuss 
and  evaluate  issues  regarding  data  dictionaries,  data  recording  and  data 
reduction.  If  an  issue  cannot  be  resolved  during  the  meeting,  an  action  item  is 
assigned  to  collect  the  information  required  for  the  ACB  to  make  an  informed 
decision  on  the  importance  of  the  issue  and  the  potential  risk  that  it  may  cause 
the  program.  The  action  items  are  tracked  by  the  ACB  until  closed  and  the 
status  is  reported  to  the  program  manager.  The  combination  of  meeting  minutes 
and  action  items  provide  a  comprehensive  list  of  all  data  dictionary,  recording 
and  reduction  issues  addressed  by  the  ACB. 

2.  Team  Effectiveness 

As  part  of  this  research  effort,  a  review  of  team  effectiveness  practices 

was  conducted.  While  the  R2D2  ACB  is  a  technical  working  group,  at  a  practical 

level,  it  must  be  able  to  consider  the  policy  implications  of  the  issues  to  be 
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presented  to  the  program  manager.  The  selection  of  A  Practical  Guide  for  Policy 
Analysis  by  Eugene  Bardach  was  chosen  to  provide  some  insight  and  tools  for 
this  reality.  This  portion  of  the  literature  review  details  the  eightfold  path  problem 
solving  method.  (Bardach,  2000,  p.  1-46)  The  method  steps  are: 


1.  Define  the  Problem.  Defining  the  problem  should  be  evaluative 
and  quantifiable.  Find  conditions  that  cause  problems.  Look  for 
any  opportunities  that  are  being  missed.  Make  sure  that  to  avoid 
fitting  a  predefined  solution  into  the  defined  problem  and  be 
skeptical  when  cause  and  effect  are  presented.  (Bardach,  2000,  p. 
1-6) 

2.  Assemble  some  Evidence.  Think  about  the  value  of  the  evidence 
before  spending  resources  to  collect  it.  Conduct  a  literature  review 
and  survey  best  practices  to  determine  if  any  apply  to  this  problem. 
(Bardach,  2000,  p.  7-12) 

3.  Consider  Alternatives.  Model  the  system  and  stay  focused  on  the 
problem  as  alternatives  are  evaluated.  (Bardach,  2000,  p.  12-19) 

4.  Select  Criteria.  When  selecting  criteria,  think  about  judging  the 
outcomes  and  determine  how  to  weigh  aspects  such  as  efficiency, 
acceptability,  improvement  and  robustness.  (Bardach,  2000,  p.  19- 
26) 

5.  Project  the  Outcomes.  Projection  =  Model  +  Evidence.  Estimate 
the  magnitude  of  the  outcome  and  consider  where  is  the  break¬ 
even  point.  Don’t  be  too  optimistic  and  remember  to  consider  the 
undesirable  side  effects.  (Bardach,  2000,  p.  27-36) 

6.  Confront  the  Trade-offs.  If  a  good  job  is  done  is  step  5,  there 
should  be  plenty  of  good  choices  of  outcomes  to  try  to  optimize  for 
a  solution.  (Bardach,  2000,  p.  37-39) 

7.  Decide.  If  the  results  of  these  steps  indicate  that  the  project  is  not 
worth  doing,  then  go  back  and  revisit  steps  1  through  6.  Just 
because  it  hasn’t  been  done  before  doesn’t  mean  it  isn’t  a  good 
solution  even  if  it  seems  too  obvious.  As  Bardach  wrote,  “If  your 
favorite  policy  alternative  is  such  a  great  idea,  how  come  it’s  not 
happening  already?  The  most  common  source  of  failure  on  this 
test  is  neglecting  to  consider  the  resistance  of  bureaucratic  and 
other  stakeholders  in  the  status  quo”.  As  changes  are  identified  in 
a  project,  the  bureaucratic  and  other  stakeholders  must  be 
considered  and  how  to  achieve  their  buy-in  to  avoid  and  overcome 
resistance.  (Bardach,  2000,  p.  40-41) 
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8.  Tell  your  story.  Consciously  avoid  some  of  the  common  pitfalls 
which  include:  following  the  eightfold  path  without  thinking  it 
through,  compulsive  qualifying,  show  your  work  but  stay  focused  on 
what  is  really  important,  listing  without  explaining  will  not  help  the 
readers  and  evaluators  to  understand  your  decisions,  avoid  a 
pompous  insider’s  style.  (Bardach,  2000,  p.  41-46) 

3.  Project  Management 

The  R2D2  ACB  requires  careful  consideration  of  the  management  of 
projects.  Visualizing  Project  Management  by  K.  Forsberg,  H.  Mooz,  and  H. 
Cotterman  (2000)  provides  the  following  practical  advice  in  project  management. 
Eliminating  features  from  a  proven  template  must  be  justified  and  to  proactively 
manage  a  project  requires  an  approach  that  is  “orderly,  methodical  and 
disciplined”.  (K.  Forsberg,  H.  Mooz,  and  H.  Cotterman,  2000,  p.  77)  During 
Combat  System  Ship  Qualification  Trials  (CSSQT),  the  analysis  of  weapon 
system  performance  requirements  must  be  balanced  with  the  computer  program 
development  resources  and  the  test  and  evaluation  schedule  before  any 
changes  are  requested  to  the  test  events.  The  ACB  is  cognizant  about  trying  to 
gain  efficiency  at  the  expense  of  eliminating  a  necessary  function.  The  ACB 
function  of  conducting  analysis  acts  as  a  control  gate  with  reporting  to  the 
program  office  and  numerous  System  Engineering  Councils  (SEC). 

There  are  three  aspects  to  the  project  cycle:  business,  budget,  and 
technical.  (K.  Forsberg,  H.  Mooz,  and  H.  Cotterman,  2000,  p.  87-88)  A  template 
for  a  project  cycle  that  one  can  compare  to  a  list  of  events  is  given  in  the  next 
figure.  It  is  unwise  to  allow  the  pursuit  of  ‘better,  cheaper,  and  faster’  to  allow  the 
discarding  of  controls  and  elimination  of  tasks  without  regard  to  the 
consequences.  “The  key  to  success  is  to  design  a  tailored,  gated  cycle  that  is 
based  on  a  proven  template  but  that  is  lean,  efficient,  and  effective.”  (K. 
Forsberg,  H.  Mooz,  and  H.  Cotterman,  2000,  p.  107) 
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4. 


Types  of  Testing 


In  the  acquisition  cycle  for  the  AEGIS  Weapon  System,  many  different 
levels  of  testing  are  performed.  Based  upon  the  past  R2D2  ACB  efforts,  the 
phases  of  testing  that  have  been  of  particular  importance  to  the  R2D2  ACB  are: 


1.  Land  Based  Computer  Program  Demonstration  Test  -  A  formal 
test  that  is  performed  to  demonstrate  computer  program 
functionality  to  demonstrate  compliance  to  the  top-level 
specification. 

2.  System  Program  Certification  Test  -  Combat  system  certification 
evaluates  a  computer  programs  ability  to  perform  required  ship 
missions  in  accordance  with  specifications,  validates  computer 
programs  do  not  regress  from  the  capability  of  the  programs  being 
replaced,  verifies  the  computer  programs  are  stable  in  use  and  that 
the  computer  programs  can  be  safely  operated  under  normal 
operating  conditions. 

3.  Combat  System  Ship  Qualification  Trial  -  A  set  of  at-sea  test 
and  evaluation  events  to  demonstrate  weapon  system  functionality 
and  the  ability  to  perform  required  ship  missions  in  accordance  with 
specifications  on  a  specific  surface  ship  platform. 

4.  Developmental  Test  (DT)  -  Testing  to  determine  technical 
performance  and  specification  compliance. 

5.  Operational  Test  (OT)  -  Testing  to  determine  the  operational 
effectiveness  of  the  system 

Figure  4  is  taken  from  the  Program  Managers  Toolkit  and  provides  a  very 
succinct  description  of  the  Test  and  Evaluation  phase. 
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Figure  4.  Test  and  Evaluation 

(DAU  Program  Managers  Tool  Kit,  2003,  p.51) 


5.  Risk  Management  and  Acquisition  Planning 

As  part  of  the  literature  review,  some  key  points  and  concepts  found  in  the 
Software  Engineering  Institute’s  Team  Risk  Management  Model  and  the 
Guidelines  for  Successful  Acquisition  and  Management  of  Software-Intensive 
Systems  (GSAM)  are  provided.  Risk  management  is  a  critical  part  of  every 
acquisition  program.  An  effective  risk  management  approach  will  identify  and 
plan  for  areas  of  risk  in  a  program  and  analyze  those  risks.  Once  this  is 
accomplished,  a  plan  to  monitor  and  control  the  areas  of  risk  can  be  put  into 
place.  To  effectively  manage  risk  in  an  optimal  program,  trade-offs  between 
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cost,  performance  and  schedule  will  need  to  be  performed.  Risk  management 
has  been  defined  as  “a  discipline  and  environment  of  proactive  decisions  and 
actions  to 

1 .  assess  continuously  what  can  go  wrong  (risks). 

2.  determine  what  risks  are  important  to  deal  with. 

3.  implement  strategies  to  deal  with  those  risk.” 

(Higuera,  R,  Dorofee,  A,  Walker,  J,  Williams,  R,  1994,  p.  6), 

At  the  center  of  effective  risk  management  is  communication.  Without 
effective  communication,  the  impact  of  an  area  of  risk  cannot  be  adequately 
assessed  and  dealt  with.  For  risk  management  to  be  effective,  it  takes  more  than 
just  the  Program  Manager.  At  all  levels  of  the  program  and  every  phase  from 
design  to  development  to  production,  risks  must  be  identified  and  mitigated 
where  possible.  When  a  risk  cannot  be  effectively  mitigated  due  to  resource  or 
other  constraints,  it  should  be  documented  to  ensure  that  the  program  cost, 
schedule  and  performance  requirements  are  effectively  managed.  (GSAM,  2000, 
p.  6-10)  The  intent  is  to  integrate  risk  management  into  the  program  team  efforts 
and  not  allow  risks  to  proceed  into  a  program  undetected.  As  demonstrated  in 
Figure  5,  risk  management  requires  trading  off  estimated  cost  versus  potential 
losses. 


Figure  5.  Cost/Benefits  of  Effective  Risk  Management 

(GSAM,  2000,  p.6-6) 
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Risk  mitigation  is  an  important  part  of  risk  management.  In  this  process, 
risk  is  identified  and  evaluated.  Once  this  is  completed,  options  are  considered 
to  set  the  identified  risks  at  an  acceptable  level  within  the  resource  constraints.  A 
critical  part  of  this  process  includes  assessing  the  possible  consequences  of 
inaction.  Ignoring  predictable  risks  is  a  common  problem  that  the  program 
manager  needs  to  be  aware  of.  As  each  risk  item  proceeds  through  the  risk 
management  process,  the  decision  on  how  to  deal  with  each  risk  item  must  be 
communicated  to  the  program  manager.  Without  authorization  from  the  PM, 
resources  cannot  be  allocated  and  preventative  action  cannot  be  undertaken. 
One  risk  that  is  true  for  all  acquisition  programs  is  funding.  Ensuring  that  “a  well- 
defined  set  of  requirements  and  active  management  involvement”  can  help  to 
mitigate  this  area  of  risk.  The  levels  of  risk  are  listed  in  Table  1 . 
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Magnitude 

Guidelines 

Critical 

High  likelihood  of  severely  impacting 

one  or  more  factors,  i.e.,  cost  & 

schedule,  performance,  or 

supportability. 

High 

High  likelihood  of  moderately  impacting 

one  or  more  factors. 

Medium 

Medium  likelihood  of  moderately 

impacting  one  or  more  factors. 

Low 

Low  likelihood  of  moderately  impacting 

one  or  more  factors. 

Table  1.  Level  of  Risk  Management  and  Guidelines 

(GSAM,  2000,  p.6-41) 

Software  development  programs  can  possess  many  subtle  environmental 
risk  factors  (GSAM,  2000,  p.6-36).  Software  developments  can  be  very  complex. 
Integration  of  different  modules  can  result  in  complex  interactions  that  are  not 
easily  resolved.  Problem  elements  have  multidimensional  relationships.  Adding 
more  people  to  a  project  may  not  increase  effectiveness.  In  fact,  it  may  reduce 
the  productivity  of  the  team.  Software  problems  can  change  and  are  inherently 
unstable.  Actual  costs  and  time  to  develop  are  difficult  to  accurately  project. 
Software  development  is  dynamic.  Due  to  changes  in  the  requirements  and 
resources,  the  development  progress  and  environment  will  continuously  change. 

Software  development  requires  people  who  represent  a  major  source  of 
risk.  Conflicts  in  human  environment,  interaction  and  desires  will  cause 
problems  since  software  development  is  a  “very  human  endeavor”  (GSAM,  2000, 
p.6-36).  Software  development  requires  performing  up-front,  strategic  planning 
to  address  problems  that  will  be  more  costly  to  fix  in  the  later  phases.  In  fact, 

21 


“The  time  you  spend  defining  your  acquisition  strategy  early  on  will  go  a  long  way 
in  assuring  stability  throughout  the  entire  life  of  the  system...”  (GSAM,  2000,  p.6- 
36)  The  use  of  tools  such  as  work  breakdown  structures  can  assist  in  linking  the 
strategic  goals  with  the  software  development  efforts.  The  use  of  open  systems 
is  expected  to  improve  the  supportability  of  software  systems.  Some  classic 
mistakes  in  software  acquisition  include  unrealistic  estimates  in  resources 
required  for  development,  inadequate  software  test  and  integration,  significant 
code  development  before  requirements  are  stable. 


22 


III.  R2D2  ACB  CASE  STUDY 


A.  CASE  STUDY  METHODS  -  PAST  PERFORMANCE  -  WHAT  HAS  THE 

R2D2  DONE? 

The  R2D2  ACB  meetings  are  generally  conducted  using  a  Video 
Teleconference  format.  This  medium  has  been  selected  to  maximize 
stakeholder  participation  and  interaction.  Early  on,  it  was  determined  that,  due  to 
the  locations  of  the  stakeholders,  it  would  be  difficult  to  get  a  consensus 
participation  in  one  location.  This  ruled  out  the  traditional  announce  a  meeting 
and  have  everyone  meet  at  one  location,  typically  the  Washington  Navy  Yard 
(WNY).  Many  of  the  issues  that  the  ACB  had  to  deal  with  have  been  complex. 
Discussing  complex,  data  related  issues;  over  a  voice  only  teleconference  was 
determined  to  be  ineffective  for  understanding  many  of  the  issues.  Email  and 
electronic  discussion  boards  were  considered  and  were  determined  to  be  useful 
as  tools,  but  not  as  the  primary  method  of  interaction. 

The  use  of  video  teleconferencing  has  maximized  participation  from  all 
stakeholders.  The  video  teleconference  format  allows  participation  from  the 
activity  site  and  is  the  least  disruptive  option  for  stakeholder  participation 
compared  to  on  site  meetings  at  WNY.  For  a  typical  meeting,  there  are  3  to  4 
nodes.  A  node  is  a  site  that  has  a  VTC  connection.  For  example,  the  R2D2  ACB 
meeting  in  29  April  2004  had  3  nodes.  The  nodes  were  at: 


•  NGMS,  Washington  DC:  80  M  Street  SE,  Suite  500  Chesapeake  Room 
5123 

•  NSWC  Corona:  Building  511,  Room  109A 

•  LM  MMS,  Moorestown,  NJ:  4000  Building  Room  374 

Email  communication  is  essential  to  the  effective  operation  of  the  R2D2 
ACB  and  meeting  announcements  are  promulgated  using  this  method.  Email  is 
frequently  where  issues  are  first  socialized  to  determine  if  their  importance  and 
level  of  program  impact  merits  discussion  during  a  meeting. 
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Another  tool  that  is  used  by  the  ACB  is  the  Corporate  Document 
Management  System  (CDMS).  CDMS  allows  all  documents  releasable  by  the 
R2D2  ACB  to  be  accessed  by  all  participants.  This  has  provided  many  benefits. 
The  primary  benefit  is  that  large  files  such  as  presentations  from  meetings  do  not 
need  to  be  emailed  to  all  participants.  Many  participants  are  only  interested  in  a 
subset  of  the  issues  discussed  by  the  ACB  and  CDMS  allows  each  participant  to 
retrieve  only  in  the  information  that  interests  them.  An  additional  benefit  is  that 
participants’  mailboxes  would  not  be  filled  up  with  large  files  from  the  ACB 
meetings  or  information  distributions.  Another  benefit  provided  by  CDMS  is  that 
it  provides  a  historical  record  of  information  on  topics  discussed.  When  a  new 
member  joins  the  R2D2  ACB  as  a  participant,  CDMS  can  be  accessed  to  find 
previous  information  on  topics  of  interest,  allowing  a  new  member  to  learn  quickly 
about  topics  of  interest. 

The  R2D2  ACB  has  assigned  a  total  of  194  action  items  since  its 
inception.  Of  these  action  items,  only  18  action  items  remain  open  as  of  15 
August  2004. 


B.  SURVEY  -  FORMATS  AND  METHODOLOGY 


Various  methods  were  considered  for  a  survey  to  gather  information  to 
answer  the  research  questions  posed  in  this  thesis.  Options  for  the  survey 
included  (University  of  Phoenix,  2002,  p.  271-280): 


Multiple  Choice  and  True/False  questions  -  Multiple  choice  and  True  or 
False  questions  were  not  used  since  they  would  imply  a  known  set  of 
answers.  These  types  of  responses  are  unlikely  to  provide  insight  into 
future  directions.  An  objective  of  this  research  is  to  determine  the 
likelihood  of  success  for  potential  future  opportunities  and  which  directions 
are  likely  to  be  successful. 

Ad  Hoc  questions  -  Ad  Hoc  method  where  questions  are  improvised 
during  the  survey  or  posed  only  of  certain  survey  participants  were  not 
selected.  The  analysis  of  the  data  from  an  Ad  Hoc  survey  is  problematic 
as  similar  questions  may  be  asked  but  the  differences  are  subtle  enough 
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to  preclude  having  the  data  correlated.  Ad  Hoc  surveys  can  lead  to  small 
sample  sizes  as  most  questions  are  unique  and  results  cannot  be  easily 
combined. 

Open-ended  questions  -  Open-ended  questions  were  not  used  for  the 
survey.  Open-ended  questions  do  not  provide  a  standardized  set  of 
responses  that  can  be  easily  correlated. 

Closed-ended  questions  -  Closed-ended  questions  were  used  for  the 
survey.  Since  the  survey  was  distributed  by  electronic  format  and  follow¬ 
up  was  conducted  in  person  or  by  telephone,  standardization  of  responses 
was  a  critical  consideration. 

Based  upon  discussions  with  the  ACB  Co-Chair,  Geoff  Uy,  and  ACB 
member  Michelle  Gallagher,  the  five  questions  were  chosen  for  this  survey.  The 
questions  were  designed  to  provide  insight  into  the  success  of  past  efforts  and 
likelihood  of  success  future  opportunities  for  the  R2D2  ACB.  (University  of 
Phoenix,  2002,  p.  271-280): 


•  How  effective  in  providing  value  to  the  AEGIS  program  has  the  R2D2  ACB 
been? 

•  How  effective  do  you  expect  the  R2D2  ACB  to  be  in  AEGIS  Open 
Architecture  baselines? 

•  How  would  you  characterize  the  amount  of  time  you  are  allowed  to  support 
R2D2  issues? 

•  At  your  activity,  how  well  does  the  management/leadership  understand  and 
support  your  involvement  in  the  R2D2  ACB? 

•  How  much  benefit  do  you  think  the  R2D2  ACB  concept  would  provide  to  a 
new  weapon  system  program? 

The  following  choices  for  possible  answers  to  each  question  were 

provided  as  follows: 

1 .  Excellent  or  Extremely  Effective 

2.  Good  or  Very  Effective 

3.  Fair  or  Effective 

4.  Poor  or  Ineffective 

5.  No  Opinion 
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The  selection  of  these  choices  will  allow  for  a  quantitative  and  qualitative 
analysis.  (University  of  Phoenix,  2002,  p.  271-280) 


C.  OPEN  ARCHITECTURE 

Open  architecture  is  a  concept  that  is  being  put  into  practice  that  allows 
for  information  to  flow  across  boundaries  and  allow  for  interoperability  between 
systems  or  components.  The  OPEN  Group  defines  the  characteristics  of 
information  flow  across  system  boundaries  as: 

“It  has  open  standard  components  that  provide  services  in  a  customer's 
extended  enterprise  that:  Combine  multiple  sources  of  information,  Deliver 
information  to  the  places  where  that  information  is  needed,  and  In  the  right 
context  for  the  people  or  systems  using  that  information.  “(Blevins,  T.,  2004, 1J24) 

Designing  open  systems  is  a  challenge  not  only  to  the  commercial 
industry  but  for  military  systems  as  well.  The  Open  Systems  Joint  Task  Force 
has  defined  Open  Systems  as  “A  System  That  Implements  Sufficient  Open 
Specifications  for  Interfaces,  Services,  and  Supporting  Formats  to  Enable 
Properly  Engineered  Components  to  be  Utilized  Across  a  Wide  Range  of 
Systems  With  Minimal  Changes,  to  Interoperate  With  Other  Components  on 
Local  and  Remote  Systems,  and  to  Interact  With  Users  in  a  Style  That  Facilitates 
Portability.”  (Strei,2003,  p.  6)  The  R2D2  ACB  is  chartered  to  ensure  that  the 
information  that  is  collected  for  combat  weapon  systems  such  as  the  AEGIS 
platform  allows  for  a  reconstruction  of  events  involving  the  weapon  system.  The 
reconstruction  of  events  is  not  limited  to  test  and  evaluation  but  extends  to 
events  occurring  during  ship  operations  including  ship  deployments  and  the  data 
recording  and  extraction  capability  to  support  the  performance  assessment  of  the 
combat  weapon  system. 

There  are  two  parts  to  the  OA  Transformation: 

1.  The  OA  Transformation  Roadmap  is  designed  to  quickly  rollout 
Navy  Open  Architecture  (NOA). 
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2. 


The  Rapid  Capability  Insertion  Process/Advanced  Processor  Build 
(RCIP/APB)  to  facilitate  the  insertion  new  capability  within  the  Open 
Architectures  as  they  are  developed  and  become  available. 


(Rushton,  W.  CAPT,  USN  et  al,  2004,  p.  v) 
There  are  a  number  of  platforms  that  are  programmed  for  Open 
Architecture  including  the  DD(X)  Multi-Mission  Destroyer  and  the  CG/DDG 
AEGIS  platforms  as  shown  in  Figure  6.  With  a  large  number  of  platforms 
converging  to  open  architecture,  the  data  recording  and  reduction  requirements 
from  one  platform  is  likely  to  impact  other  platforms  as  these  open  systems  are 
integrated  together  to  meet  the  objectives  of  Network  Centric  Warfare  through 
the  implementation  of  FORCENET.  While  the  introduction  of  COTS  hardware 
has  been  implemented  in  some  cases,  OA  will  bring  commercial  software 
structures  and  designs  into  combat  weapon  systems  to  support  the  COTS 
hardware  and  complete  the  replacement  of  obsolescent  MIL-STD  systems. 


Figure  6.  OA  Transformation  Roadmap 

(Rushton,  W.  CAPT,  USN  et  al,  2004,  p.  29) 
27 


The  two  programs  that  are  specifically  being  researched  in  this  study  are: 


•  AEGIS  Open  Architecture  (AOA) 

•  DD(X) 

Both  of  these  platforms  are  Naval  Surface  Combatants  that  have  been 
required  to  implement  architecture  that  is  compliant  with  Navy  Open  Architecture 
(NOA)  standards. 


AWS  COMPUTER  ARCHITECTURE 
EVOLUTION  PATH  AFTER  PQM-04 


Figure  7.  AEGIS  Baseline  Architecture 

(Williams,  J.,  2003,  p.  4) 


The  development  of  AWS  has  been  incremental  and  evolutionary  over 
time.  As  shown  in  Figure  7,  early  AEGIS  baselines  were  MIL-STD  equipment 
and  COTS  introduction  began  with  Baseline  6  Phase  1 .  A  demonstration  test  of 
the  first  all  COTS  AEGIS  Baseline,  7  Phase  1  was  completed  in  August  2003. 
AEGIS  Baseline  7  Phase  1C,  the  first  AEGIS  baseline  to  implement  AEGIS  Open 
Architecture,  is  scheduled  for  demonstration  testing  in  2005.  AEGIS  Baseline  7 
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Phase  1C  is  accomplishing  this  through  the  use  of  evolutionary  and  spiral 
development.  There  are  3  spirals  planned  for  AEGIS  Baseline  7  Phase  1C. 
AOA  is  being  implemented  in  5  spirals  as  shown  in  Figure  8.  The 
implementation  of  this  architecture  is  spread  across  three  areas  that  are:  Radar 
Control,  Weapons  Control,  and  Display. 
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Figure  8.  AEGIS  Open  Architecture  Spirals 

(System  Engineering  Management  Plan  (SEMP)  For  AEGIS  CG/DDG 
Open  Architecture  (Draft),  2003,  p.  11) 


Through  the  use  of  evolutionary  and  spiral  development,  AEGIS  will  be 
able  to  build  a  little  and  then  test  the  result.  This  strategy  will  be  critical  to  the 
ultimate  success  of  AOA  since  the  adaptation  of  the  system  is  occurring  in 
spirals.  The  spiral  development  process  presents  challenges  in  the  data 
dictionary,  recording  and  reduction  arena  as  well.  As  functionality  is  migrated  to 
open  architecture,  the  challenge  is  to  ensure  that  the  information  required  to 
evaluate  the  system  is  adequate  and  available. 
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D.  DD(X)  NEXT  GENERATION  DESTROYER 


The  DD(X)  multi-mission  destroyer  program  is  being  developed  to  meet 
the  NOA  standards.  The  DD(X)  platform  is  attempting  to  pursue  advanced  Open 
Architecture  concepts  including  the  Total  Ship  Computing  Environment  (TSCE) 
and  modular  design.  As  NOA  has  a  Network  Centric  Warfare  (NCW) 
implementation,  the  validation  of  data  as  it  travels  from  one  element  to  another 
element  will  be  a  critical  component  of  the  success  of  this  weapon  system. 
Figure  10  shows  the  transformational  components  in  the  DD(X)  multi-mission 
destroyer  which  is  designed  for  littoral  and  network  centric  warfare. 


Figure  9.  DD(X)  Multi-Mission  Destroyer 

(DD(X)  Multi-Mission  Destroyer,  2004) 
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Figure  10.  Breakout  of  the  DD(X)  Transformational  Systems. 


(DD(X)  Transformational  Systems,  2004) 


The  initial  design  includes  the  following  transformational  systems  (DD(X) 
Transformational  Systems,  2004,  Ifl)  that  may  require  data  recording  and 


reduction: 


•  Total  Ship  Computing  System  -  The  Local  Area  Network 
responsible  for  integrating  warfare  capability  within  the  weapon 
system  platform.  TSCE  will  provide  a  Common  Operating  Picture 
(COP)  by  fusing  together  data  provided  by  the  onboard  sensors 
and  systems  and  data  received  from  external  sources.  TSCE  is 
designed  to  the  requirements  of  NOA. 

•  Advanced  Gun  System  -  Designed  to  provide  greater  firepower 
while  being  unmanned.  Provides  a  land  strike  capability  through 
the  ability  to  fire  advanced  munitions  and  propelling  charges.  A 
GPS  guided  Long  Range  Land  Attack  Projectile  (LRLAP)  is  one  of 
the  expected  projectiles  to  be  developed  for  this  system. 

•  Integrated  Undersea  Warfare  -  Designed  to  provide  In-stride  Mine 
Avoidance  using  a  dual  (HF/MF)  frequency  bow  array 
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•  Peripheral  Vertical  Launch  System  -  Designed  to  launch 
STANDARD  SM-2  and  Evolved  Sea  Sparrow  Missiles  that  are 
placed  in  a  modularly  designed  launcher.  By  placing  the  launchers 
along  the  perimeter  of  the  ship,  reduces  the  loss  of  cells  from  a 
single  hit. 

•  Dual  Band  Radar  -  Integrates  two  active  phased-array  radars  for 
detection  of  potential  threats  and  fire  control  illumination  during  the 
engagement  of  selected  threats.  DBR  will  integrate  the  L-Band 
Volume  Search  Radar  with  the  X-Band  Multifunction  Radar  to 
perform  search  and  track  performance. 


Figure  11.  Components  of  the  Dual  Band  Radar. 

(Dual  Band  Radar,  2004) 


As  the  design  of  combat  weapon  systems  such  as  the  DD(X)  becomes 
more  automated  in  the  pursuit  of  reduced  manning  levels,  more  information  will 
need  to  be  captured  to  ensure  that  system  responses  to  internal  and  external 
stimuli  can  be  reconstructed.  As  the  first  built-from-scratch  open  architecture 
combat  weapon  system,  the  data  recording  philosophy  and  structure  will  be 
carried  into  many  future  combat  weapon  systems  as  modules  from  this  platform 
are  reused  in  future  systems.  The  development  of  the  DD(X)  platform  as  a 
network  centric  platform  will  provide  opportunity  to  use  open  architecture 
components  for  other  platforms  on  the  OA  Transformation  Roadmap  (Figure  6). 
Assuming  a  complete  level  of  conformance  to  open  architecture  requirements, 
functional  modules  from  one  weapon  system  could  be  easily  moved  to  another 
system.  (Rushton,  W.  CAPT,  USN  et  al,  2004,  p.  29-32) 
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E. 


TRENDS  IN  DEFENSE 


As  Secretary  of  Defense,  The  Honorable  Donald  Rumsfeld,  stated  on  4 
June  2004,  “If  the  Department  of  Defense  is  to  stay  prepared  for  the  security 
challenges  of  the  21st  century,  we  must  transform  not  just  our  defense 
strategies,  not  just  our  military  capabilities,  not  just  the  way  we  deter  and 
defend — but  we  must  also  transform  the  way  we  conduct  our  business.”  (Glaros, 
2003,  p.  1 )  The  R2D2  ACB  must  consider  how  this  transformation  will  affect  Test 
and  Evaluation  Events  and  areas  within  the  R2D2  ACB  charter.  Since  the  R2D2 
ACB  has  been  responsible  for  Naval  Weapons  Systems,  the  Navy’s  vision 
outlined  in  SEA  POWER  21  will  be  the  guiding  document  for  determining  trends 
in  defense  that  the  R2D2  ACB  should  align  to. 


SEA  POWER  21 


Figure  12.  SEA  POWER  21 

(Clark,  2002,  p.  1) 


In  SEA  POWER  21,  the  ability  to  share  information  is  discussed.  SEA 

POWER  21  states  “Once  information  is  acquired,  it  must  be  shared  and 

processed  to  achieve  knowledge  dominance,  leading  to  operational  advantage. 

To  meet  this  challenge,  the  Navy  has  been  improving  data  sharing  among 
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platforms,  including  Link  16,  the  Cooperative  Engagement  Capability... ’’(Mayo  & 
Nathman,  2003,  p.  2).  Link  16  and  Cooperative  Engagement  Capability  (CEC) 
are  an  integral  part  of  the  AWS  and  will  be  part  of  DD(X).  As  information  is 
shared  across  many  platforms,  the  data  will  become  part  of  the  system.  NCW 
will  allow  the  integration  of  the  battlegroup  to  respond  to  threats  more  quickly. 
SEA  POWER  21  “will  require  new  models  for  command,  control,  and  data  flow.” 
(Mayo  &  Nathman,  2003,  p.  1)  resulting  in  the  need  to  integrate  data  from  many 
platforms  to  understand  system  performance.  SEA  POWER  21  will  provide  the 
warfighter  with  a  complex  set  of  data  on  which  to  base  responses  to  possible 
threats.  In  this  changing  environment,  “rapid  information  collection,  analysis, 
dissemination,  decision  making,  and  execution  are  critical  to  winning  the  life-and- 
death  race  for  combat  effectiveness.  Swift  and  effective  use  of  information  will 
be  central  to  the  success  of  SEA  POWER  21 ."  (Mayo  &  Nathman,  2003,  p.  1 ) 

Part  of  SEA  POWER  21  discusses  the  concept  of  Sea  Basing  that  allows 
the  Navy  to  be  on  the  scene  and  in  theater  wherever  needed.  To  meet  this 
objective,  the  Navy  is  relying  on  new  systems  that  are  being  designed  to  open 
architecture  standards.  These  systems  include  the  CVN(X)  nuclear-powered 
aircraft  carrier  and  the  multi-mission  DD(X)  destroyer.  The  CVN(X)  and  DD(X) 
platforms  will  have  significantly  lower  manning  levels  than  previous  generation 
aircraft  carriers  and  destroyers.  To  reduce  manning  levels,  more  autonomy  of 
the  systems  on  the  ship  will  be  required,  including  the  combat  system.  As  there 
is  less  man-in-the-loop,  validity  of  data  in  the  system  will  become  more  critical  to 
ensuring  the  quick  and  appropriate  response  to  imminent  threats.  As  advanced 
weapons  and  sensors  are  netted  together,  the  requirements  for  data  processing 
and  transfer  will  continue  to  grow  exponentially. 

Vice  Admiral  Albert  Konetzni,  the  Deputy  Commander  and  Chief  of  Staff 
for  the  Atlantic  Fleet  discussed  the  need  for  “solid  intellectual  analysis”  (Costa, 
2003,  p.  19)  while  transforming  our  defense  capability.  He  discussed  the  lack  of 
rigor  in  operation  analysis  in  favor  of  stop  light  charts  in  PowerPoint 
presentations  without  substantive  analysis  to  support  it.  (Costa,  2003,  p.  19)  The 

R2D2  ACB  has  provided  a  forum  for  supporting  rigorous  analysis  of  test  events. 
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The  R2D2  ACB  has  worked  to  ensure  that  the  data  required  for  analysis  of  the 
AEGIS  weapon  system  is  designed  into  the  system.  VADM  Konetzni 
recommends  “using  realistic  scenarios  and  experiments  to  make  sure  they  will 
work  as  advertised”.  (Costa,  2003,  p.  20)  By  ensuring  that  the  data  required  for 
analyzing  these  tests,  the  R2D2  ACB  is  working  toward  this  goal. 

The  importance  of  incorporating  modeling  and  simulation  to  reduce  the 
reliance  on  live  fire  testing  is  discussed  in  the  Chief  of  Naval  Operations  (CNO) 
Guidance  for  2004,  Accelerating  Our  Advantages.  The  use  of  modeling  is 
consistent  with  the  reduction  in  live  fire  events  that  has  been  evident  in  the 
AEGIS  program  in  recent  years.  The  use  of  modeling  is  consistent  with  the  Sea 
Enterprise  initiative  to  “leverage  technology  to  improve  performance”.  (CNO 
Guidance  for  2004,  Accelerating  Our  Advantages,  2003,  p.  20)  The  AEGIS 
program  has  a  history  of  using  land  based  testing  and  simulated  testing  on  the 
ship  platform.  The  resolution  of  track  identification  in  a  joint  environment  will 
continue  to  present  problems  in  assessing  combat  weapon  system  performance. 
The  reduction  of  live  fire  test  and  evaluation  events  will  require  the  collection  and 
analysis  of  data  from  modeling  and  simulation  combined  with  data  from  events  at 
test  ranges,  land  based  test  sites  and,  all  other  available  test  facilities  in  order  to 
effectively  evaluate  system  performance. 
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IV.  RESEARCH  ANALYSIS  AND  APPLICATION 


A.  INTRODUCTION 


The  design  of  the  R2D2  ACB  is  intended  to  be  similar  to  an  Integrated 
Product  Team  that  is  limited  to  the  areas  affected  by  data  recording.  The  ACB 
has  a  cross  section  of  system  designers  and  users.  In  this  case,  the  users  are 
the  analysts  who  require  data  to  perform  their  function.  While  IPT’s  have 
authority  to  make  decisions  within  the  scope  of  their  effort,  the  R2D2  ACB 
collects  information  and  informs  the  program  manager  on  data  dictionary, 
recording  and  reduction  issues.  The  ACB  does  not  have  the  authority  to 
implement  a  decision.  The  R2D2  ACB,  unlike  some  IPT’s,  does  not  choose  its 
membership.  Each  activity  assigns  the  people  that  they  desire  to  support  this 
effort  and  can  remove  or  change  members  based  upon  changing  priorities  within 
the  activity.  This  has  resulted  in  the  loss  of  some  of  the  most  productive 
members  of  the  ACB;  however,  membership  changes  have  provided  the 
opportunity  to  find  new  champions  for  the  R2D2  ACB  within  their  respective 
organizations. 

The  involvement  of  the  R2D2  ACB  has  changed  as  recent  AEGIS 
baselines  have  been  introduced.  In  AEGIS  Baseline  6  Phase  1,  the  R2D2  ACB 
did  not  exist  until  late  in  the  design  and  development  cycle.  Accordingly,  the 
ACB  was  able  to  assess  past  mistakes  and  try  to  determine  lessons  learned  for 
future  baseline  development.  The  roadmap  for  AEGIS  Baseline  6  Phase  3 
developments  that  were  presented  in  the  CNO  Project  801  (DDG  51  Class)  CNO 
Project  1669  (Cruiser  Modernization)  R2D2  In-Brief  is  illustrated  in  Figure  13. 


37 


NOTIONAL  ROAD  MAP  TO  801  OT-IIIG 


SHIP 

Yard  Port  (or  > 
PROGRAM 


H  I.  6  PH  3 
Me i  YMPBFJ  l 

<  a  DDG* 

B  (PMRF) 

SHOIP 
WA  l)W;84 
I  (PMRF) 
MASON** 

VA.  DDC;*7 
B  (AFWTF 
PRI  111  I 
CA.  DDG38 
I  (PMRF) 
MtSTIN 
CA.  DDG89 
I  (PMRF) 
(HART. 

m  dw;  90 

B  (PMRF) 
ESSM 

Tm  ship 
DDGM 

INTEROP 


NFC'S 
TEMP  ISPS 


2otl  2002  2M3  2**4 

FMAMi  i  A30NDJ  FMAMJ  i  A3QN  D  |  I  FMAMJ  i  A  3  O  N  DM  FMAMJ  J  A 


72T 

DEM  <X’ PAP 


<j.a 

to 


«-U 

to 


TRIALS  A  B/C 
TJll 

to 

L* 

to 


Bf,SA..4WA^^f] 

■  IWUnlA  B/(  * 

■  5  SA1L%AY  r^l 

ALS  A  WC  W 


TRIALS  AIT  C 

j  A 


TRIALS  A/B  C 


SAII-A  WAY  fj,# 


r 


TRIALS  A/B  C 


~7Vnswcdi> 

CPCP 


SCN 


O  A 

SCN  Teal  Wind**  rod 


sailaVv  l£^l[£7]  0 

•A#  AJ  SCN 


Si  J  LA  WAY 


TRIAL  i  A/ 
DTCT 


*  JL  N3  ed  « 


^^MS-III  (FRP) 

rT^n 


Jew] 


KMl^  Hi 

Juction 

1  T| 

MiEval 

Open 

M 

A  S 

N 

D 

J  F 

M  A  1 

M 

J  J  A 

J  F  M  A  M  J  i  A 


N  D  J  FMAMJ  i  AS 


N  D|J  FMAMJ  J  A 


Date*  for  tk*  Trial*,  PSA  and  SCN  Cutoff  are  baaed  on  tke  1  April  2003  DDC  SI  Claaa  HIW  MYT  Sekedule. 
CS8QT  Dnlea  nr*  baaed  on  I  May  2003  Teat  and  Evalaatioo  (TAE)  •chrdulr.  All  utkrr  dale* 
arr  baaed  ok  other  aptirce  data 

*•  Teat  Skip 

https://aegisteclKliv.na  vsea.navy.ini  I 


ITDATEDi  1  May  2003 


Figure  13.  Baseline  6  Phase  3  Development 

(Howard,  2004,  p.  5) 


The  R2D2  ACB  was  being  established  as  the  6  Phase  3  Baseline  was 
completing  design  and  preparing  to  multi-element  integration  and  test  (MEIT). 
Baseline  6  Phase  3  provided  an  opportunity  for  the  R2D2  ACB  to  effect  system 
modification  decisions  through  the  Test  Observation  Report  (TOR)  mechanism. 
The  TOR  process  provides  a  method  for  collection  and  prioritization  of  system 
performance  anomalies  detected  during  MEIT,  engineering  test  and  evaluation, 
system  and  demonstration  testing  of  the  computer  program  baseline.  TORs  are 
rated  on  a  1  to  5  scale  based  upon  the  risk  to  the  program.  The  scale  is  as 
follows  (AEGIS  Performance  Report  for  CEO  OPEVAL,  2001): 
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Priority 

Description 

1 

Mission  Failure  or  Safety  problem 

2 

Mission  Degradation 

3 

1  or  2  with  an  acceptable  workaround.  No  workarounds 
allowed  or  safety  problems 

4 

Operator  annoyance  or  nuisance 

5 

Not  visible  to  the  operator  or  no  tactical  impact  (doc 
change).  Visible  but  trivial  problem,  e.g.  misspelling  on 
aCRO 

Table  2.  TOR  Priorities 


(AEGIS  Performance  Report  for  CEC  OPEVAL,  2001) 

Data  recording  issues  were  traditionally  downgraded  to  priority  4  or  5 
since  a  ‘work  around’  could  be  provided.  This  was  appropriate  in  many  cases. 
However,  some  data  recording  issues  were  found  to  affect  the  assessment  of 
system  performance  during  CSSQT  and  DT/OT  testing.  The  ‘work  around’  that 
was  acceptable  in  a  lab  test  environment  was  not  available  during  the  formal  at- 
sea  testing.  In  the  lab,  a  test  for  a  specific  function  could  be  conducted 
repeatedly,  but  in  at-sea  testing,  the  testing  is  very  limited  due  to  the  high 
expenses  of  ranges,  targets,  ordnance,  and  test  teams.  A  TOR  Form  is  included 
in  Appendix  C. 

An  example  of  a  low  priority  observation  occurred  during  an  AEGIS 
CSSQT  test  event.  A  target  was  presented  to  an  AEGIS  destroyer  for 
engagement  to  determine  the  effectiveness  of  the  combat  weapon  system. 
When  the  target  was  ready  to  be  engaged,  the  AEGIS  Display  System  (ADS) 
displayed  incorrect  information  to  the  console  operator  resulting  in  the  operator 
choosing  not  to  engage  the  target.  The  system  displayed  information  from  when 
the  console  had  been  in  test  mode  rather  than  the  present  state  of  tactical  mode. 
While  the  state  of  the  console  being  in  test  mode  or  tactical  mode  was 
considered  an  operator  annoyance  (Priority  4)  in  the  lab,  the  impact  was  much 
different  during  actual  at-sea  testing. 

AEGIS  Baseline  7  Phase  1  provides  the  completion  of  the  re-architecting 
of  the  AWS  to  COTS  technology.  The  R2D2  ACB  was  established  and  able  to 
influence  the  program  development  at  an  earlier  stage  than  in  Baseline  6.  At  the 

time  of  this  writing,  AEGIS  Baseline  7  Phase  1  was  preparing  for  its  first  CSSQT 
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testing  with  the  USS  PINCKNEY  (DDG  91).  A  notional  illustration  of  the 
scheduled  testing  is  provided  in  Figure  14  as  presented  in  the  CNO  Project  801 
(DDG  51  Class)  CNO  Project  1669  (Cruiser  Modernization)  R2D2  In-Brief.  As  a 
result  of  earlier  involvement,  a  significant  effort  was  undertaken  to  document 
extraction  points  and  capture  information  that  had  been  lost  in  the  data 
dictionaries  as  a  result  of  the  introduction  of  COTS.  While  the  results  did  not 
provide  everything  that  the  analysis  community  desired,  significant  progress  was 
made.  Early  progress  is  critical  for  another  reason.  In  the  AEGIS  baseline 
development,  a  capture  of  the  previous  baseline  was  performed  for  the  starting 
point  of  the  next  baseline.  Changes  that  are  made  to  the  computer  program  after 
baseline  capture  must  be  reinserted  if  it  is  to  be  included  in  future  baselines. 
This  has  proven  to  be  an  expensive  and  difficult  path  to  pursue. 
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Figure  14.  AEGIS  Baseline  7  Phase  1  Development 

(Howard,  2004,  p.  7) 


AEGIS  Baseline  6  Phase  3  provides  an  excellent  example  of  why  early 
involvement  is  critical  to  getting  an  essential  data  recording  capability  into  the 
baseline.  The  ADS  displays  tracks  to  the  operator  that  are  provided  by  the  C&D 
computer  via  the  ADS  Track  Server.  During  AEGIS  testing,  it  was  noted  that 
some  tracks  did  not  appear  at  the  console  as  expected.  Review  of  the  data 
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7  Phase  1 


recording  capability  showed  that  the  tracks  could  be  found  in  the  C&D  Track 
Database  file  but  there  was  no  data  to  indicate  when  a  track  was  actually 
displayed  to  the  operator  at  an  ADS  console.  The  inability  to  determine  whether 
a  track  was  present  at  a  console  from  the  test  data  was  determined  to  be  an 
operational  issue.  Initially,  the  R2D2  ACB  was  unable  to  convince  the  system 
developer  or  program  manager  to  implement  a  change  in  the  Baseline  6  Phase  3 
computer  program  to  collect  and  record  this  information.  After  a  number  of  tests 
indicated  the  value  of  this  information,  the  Program  Manager  became  convinced 
of  the  importance  of  this  data.  The  system  developer  was  instructed  to 
determine  a  method  to  collect  this  information  and  the  associated  cost.  After  an 
agreement  was  reached,  the  ADS  Track  File  Data  Recording  (TFDR)  was 
scheduled  into  the  development  path.  Unfortunately,  this  change  was  not 
captured  in  Baseline  7  Phase  1.  It  remained  an  open  issue  as  to  whether  the 
program  manager  for  this  baseline  will  capture  this  functionality.  However,  even 
if  this  functionality  is  eventually  captured  into  Baseline  7  Phase  1,  it  was  too  late 
for  it  to  be  captured  directly  into  Baseline  7  Phase  1C  and  will  likely  be  too  late 
for  Baseline  7  Phase  1 R. 

As  the  development  of  Baseline  7  Phase  1C  begins,  the  R2D2  ACB  is 
attempting  to  be  inserted  earlier  than  in  the  Baseline  7  Phase  1  process.  Figure 
15  illustrates  the  programmed  development  schedule  for  this  baseline,  which  was 
presented  in  the  CNO  Project  801  (DDG  51  Class)  CNO  Project  1669  (Cruiser 
Modernization)  R2D2  In-Brief.  Baseline  7  Phase  1C  is  the  first  AEGIS  Open 
Architecture  Baseline  and  the  ability  for  the  R2D2  ACB  to  affect  the  development 
of  the  data  recording  remains  to  be  seen. 
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Figure  15.  Baseline  7  Phase  1C  Development 

(Howard,  2004,  p.  15) 

B.  SUMMARY  OF  INTERVIEWS  AND  SURVEYS 

A  survey  was  distributed  to  determine  the  effectiveness  of  the  R2D2  ACB 
from  the  perspective  of  the  core  participants.  The  selection  criterion  was  people 
who  regularly  attend  and  participate  in  the  ACB.  A  total  of  eighteen  people  met 
the  attendance  and  participation  criteria  and  were  surveyed.  While  there  have 
been  more  participants  in  the  history  of  the  R2D2  ACB,  the  survey  population 
represents  the  active  core  that  has  allowed  the  R2D2  ACB  to  perform  its  mission. 
Based  upon  discussions  with  Mr.  Uy,  R2D2  ACB  Co-Chairman,  the  survey 
population  was  selected  based  upon  the  following  criteria:  number  of  meetings 
attended,  knowledge  of  data  dictionaries  and  data  reduction,  area  of  technical 
expertise  and  level  of  participation  in  meetings.  The  survey  results  are  listed  in 
Table  3. 
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Excellent  or 

Extremely 

Effective 

Good  or 

Very 

Effective 

Fair  or 
Effective 

Poor  or 
Ineffective 

No 

Opinion 

How  effective  in  providing  value  to  the  AEGIS 
program  has  the  R2D2  ACB  been? 

33.3% 

55.6% 

ii.i% 

0.0% 

0.0% 

How  effective  do  you  expect  the  R2D2  ACB 
to  be  in  AEGIS  Open  Architecture 
baselines? 

22.2% 

55.6% 

22.2% 

0.0% 

0.0% 

How  would  you  characterize  the  amount  of 
time  you  are  allowed  to  support  R2D2 
issues? 

11.1% 

33.3% 

44.4% 

11.1% 

0.0% 

At  your  activity,  how  well  does  the 
management/leadership  understand  and 
support  your  involvement  in  the  R2D2  ACB? 

55.6% 

0.0% 

44.4% 

0.0% 

0.0% 

How  much  benefit  do  you  think  the  R2D2 
ACB  concept  would  provide  to  a  new 
weapon  system  program? 

77.8% 

11.1% 

11.1% 

0.0% 

0.0% 

Table  3.  Survey  Results 

Each  question  will  be  addressed  along  with  an  analysis  of  the  results  and 
discussion  of  survey  comments  provided  by  respondents  that  are  relevant  to  that 
question.  One  result  that  applied  to  the  survey  as  a  whole  was  that  no  answers 
of  ‘No  Opinion’  were  received.  As  the  criteria  included  participation,  the  level  of 
involvement  required  to  receive  the  survey  provided  a  survey  pool  that  had 
generally  produces  strong  opinions  on  each  of  the  subject  areas  addressed  by 
the  survey  questions.  Since  the  survey  population  consisted  of  active 
participants  in  the  R2D2  ACB,  this  result  was  expected. 


1.  Survey  Question:  How  effective  in  providing  value  to  the  AEGIS 
program  has  the  R2D2  ACB  been? 

The  first  question  intended  to  capture  the  perceived  value  that  the  R2D2 
ACB  has  provided.  Most  respondents  (89%)  believe  that  the  R2D2  ACB  has 
been  very  or  extremely  effective.  This  result  is  validated  by  the  fact  that  the 
R2D2  ACB  continues  to  exist.  If  the  participants  did  not  believe  that  the  R2D2 
ACB  was  effective,  they  would  elect  to  not  participate  and  find  other  activities  to 
engage  in.  Some  program  managers  and  system  developers  do  not  view  data 
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recording  as  a  critical  component  of  the  system  under  development.  The 
experience  of  the  R2D2  ACB  has  shown  that  data  recording  problems  have  been 
downgraded  in  favor  of  more  visible  tactical  system  software  issues.  (Johnson, 
2001)  The  R2D2  ACB  is  a  champion  of  the  importance  of  data  recording  to 
understand  what  needs  to  be  fixed.  By  designing  a  robust  data  recording 
capability,  system  problems  can  be  isolated  more  rapidly  with  less  testing  and 
solutions  can  be  quickly  verified.  The  introduction  of  COTS  has  allowed  methods 
for  debugging  computer  program  development  through  methods  other  than  data 
recording.  Prior  to  COTS,  the  program  developers  depended  upon  the  ATES 
data  recording  to  obtain  memory  dumps  that  were  used  to  determine  program 
faults  and  errors.  On  the  positive  side,  the  opportunity  to  bring  together  the 
program  developers,  program  managers  and  analysis  community  was  viewed  as 
beneficial  in  reducing  redundant  efforts  and  allowing  the  validation  of  problems 
and  their  importance.  The  R2D2  ACB  provided  the  ability  for  open  dialog  in 
discussing  the  issues,  mitigation,  and  planning  implementation.  With  the  diverse 
AEGIS  community,  the  ACB  allows  understanding  of  the  impact  a  change  might 
have  to  another  organization  in  near  real  time.  As  long  as  the  AEGIS  program 
continues  to  change,  there  will  be  a  need  to  test  it  and  work  though  issues  that 
will  arise  -  that  is  where  the  strength  of  the  R2D2  ACB  exists. 

2.  Survey  Question:  How  effective  do  you  expect  the  R2D2  ACB  to  be 

in  AEGIS  Open  Architecture  baselines? 

The  second  question  is  a  forward-looking  question  to  provide  insight  into 

the  R2D2  ACB  participant’s  opinion  of  the  future  for  the  ACB.  All  future 

development  for  the  AEGIS  program  is  programmed  for  the  AEGIS  Open 

Architecture  baselines  once  development  of  Baseline  7  Phase  1  is  completed.  In 

the  opinion  of  78%  of  the  survey  respondents,  Program  Manager  buy-in  is 

viewed  as  critical  to  the  success  in  future  baseline  development.  In  the  AEGIS 

OA  development,  the  use  of  Technical  Performance  Measures  (TPM)  has  been 

contractually  employed.  The  use  of  TPM’s  can  be  used  to  require  data  recording 

to  be  satisfied  providing  a  motivation  to  the  developer  to  ensure  that  specific  data 
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recording  capabilities  are  designed  into  the  system.  The  R2D2  ACB  could 
provide  the  Program  Manager  with  the  most  critical  areas  to  be  measured  and 
the  requirements  for  a  robust  data  recording  capability.  The  R2D2  ACB  would 
enable  the  data  analysis  community  to  become  proactive  in  deciding  what  needs 
to  be  measured  as  the  baseline  is  tested.  Without  a  proactive  approach,  the 
data  recording  is  likely  to  be  an  early  casualty  of  budget  and  resource  allocations 
and,  as  has  happened  in  previous  baselines,  data  recording  will  always  be 
playing  catch-up.  Finally,  the  R2D2  ACB  is  in  a  unique  position  to  see  the  data 
problems  (both  engineering  and  political)  facing  such  a  transition. 

3.  Survey  Question:  How  would  you  characterize  the  amount  of  time 
you  are  allowed  to  support  R2D2  issues? 

The  third  question  is  intended  to  provide  the  participant’s  perspective  on 
how  much  effort  they  are  able  to  provide  toward  the  R2D2  ACB  efforts.  The 
survey  results  indicated  that  over  50%  of  the  respondents  characterize  their 
ability  the  support  the  R2D2  ACB  as  fair  or  poor.  Based  upon  discussions  during 
previous  R2D2  ACB  meetings,  two  issues  that  have  limited  the  ability  of  ACB 
members  to  support  this  effort  are  funding  and  collateral  duties.  The  R2D2  ACB 
efforts  are  not  separately  identified  in  the  funding  documents.  While  the  program 
office  expects  each  activity  to  use  baseline  development  funding  for  this  effort, 
there  is  no  specific  funding  allocated  directly  for  this  effort.  When  a  conflict  in 
responsibilities  occurs,  the  bias  is  toward  the  tasks  that  are  directly  referenced  in 
funding  documents  or  statements  of  work.  Other  times,  collateral  duties  can 
overwhelm  the  work  schedule  and  result  in  R2D2  ACB  efforts  being  minimized. 

4.  Survey  Question:  At  your  activity,  how  well  does  the 

management/leadership  understand  and  support  your  involvement  in 
the  R2D2  ACB? 

The  fourth  question  is  designed  to  correlate  the  previous  question.  In 
cases  where  an  insufficient  time  is  allowed,  to  what  degree  does  this  contribute 
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to  management  being  unaware  of  the  R2D2  ACB’s  mission  and  benefits?  For 
those  who  are  satisfied  with  the  amount  of  time  available  for  R2D2  efforts,  does  it 
result  in  management  support?  Even  in  cases  where  the  management 
understands  the  importance  of  participation  in  the  R2D2  ACB,  priorities  can 
overtake  R2D2  efforts  when  direct  work  traceable  to  tasking  is  at  stake.  Another 
problem  is  that  some  people  receive  good  support  while  co-workers  receive 
somewhat  less  support  due  to  different  managers  perception  of  the  R2D2  ACB. 

5.  Survey  Question:  How  much  benefit  do  you  think  the  R2D2  ACB 

concept  would  provide  to  a  new  weapon  system  program? 

The  final  question  is  designed  to  determine  if  the  respondent  believes  that 
the  R2D2  ACB  is  unique  to  AEGIS  or  if  it  can  grow  beyond  this  program.  The 
critical  concern  is  that  the  program  managers  understand  and  champion  this 
effort.  Every  mission  is  going  to  require  a  data  recording  capability  to  measure 
its  results.  This  is  not  unique  to  the  Navy  or  DOD.  NASA  relies  on  telemetry 
data  to  determine  if  its  space  probes  are  operating  correctly  and  commercial 
aircraft  contain  “black  boxes”  that  contain  data  recording  deemed  critical  to 
reconstruct  aircraft  and  aircrew  performance.  A  significant  benefit  that  the  R2D2 
ACB  can  provide  is  combining  lessons  learned  and  guidance  on  data  recording, 
extraction,  processing  and  analysis  techniques  and  methods  to  programs  in  the 
process  of  defining  and  developing  these  capabilities. 

The  survey  results  indicate  that  78%  of  the  respondents  reported  that  the 
R2D2  ACB  efforts  would  be  extremely  effective  for  a  new  weapons  program. 
Since  the  ACB  has  been  established  for  over  3  years,  the  positive  views 
presented  in  the  survey  results  are  not  the  result  of  unguarded  optimism  for  a 
new  effort.  Rather,  it  represents  a  time-tested  reflection  upon  the 
accomplishments  of  this  forum.  By  providing  a  forum  where  issues  and 
deficiencies  can  be  discussed  without  attribution,  many  R2D2  ACB  members 
have  been  able  to  raise  issues  for  investigation  that  would  be  difficult  to  carry 
forward  at  their  activity.  The  R2D2  ACB  can  take  action  and  allow  the 
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membership  to  investigate  issues  as  an  ACB  action  rather  than  an  individual 
concern.  Furthermore,  the  management  at  an  activity  recognizes  that  action 
items  assigned  through  the  R2D2  ACB  have  been  through  a  peer  review  process 
to  determine  that  the  issue  is  valid  and  important  to  the  AEGIS  program. 

C.  R2D2  ACB  AND  RISK  MANAGEMENT 


Based  upon  the  experience  of  the  R2D2  ACB  to  date,  the  primary 
functions  that  it  performs  are  to  review  and  monitor  data  dictionary  and  recording 
issues  as  the  analysis  community  identifies  the  issues.  The  R2D2  ACB 
effectively  performs  risk  analysis  for  the  Program  Manager.  Each  issue  is 
identified  and  analyzed.  Part  of  this  process  includes  determining  the  impact  on 
the  program  performance  and  schedule.  As  the  R2D2  ACB  evaluates  issues, 
consideration  is  given  to  assess  what  information  is  lost  resulting  from  each 
specific  data  dictionary  and  recording  issue.  Next,  the  risk  resulting  from  each 
issue  is  documented  through  ACB  meeting  discussions  and  action  item 
assignments.  Once  the  issue  is  defined  and  understood,  the  analysis  community 
provides  feedback  to  determine  how  the  problem  may  be  addressed.  Once  an 
issue  is  defined  and  a  solution  identified,  the  Program  Manager  is  provided  the 
information  necessary  to  determine  whether  the  resources  required  that  would 
resolve  the  problem  is  worth  the  tradeoff  of  resources  for  other  uses. 

The  R2D2  ACB  provides  consistency  in  prioritizing  and  investigation  of 
data  dictionary  and  data  recording  issues.  Past  experience  in  the  AEGIS 
program  has  demonstrated  a  dependency  upon  the  Program  Manager.  When  a 
Program  Manager  is  concerned  about  data  dictionary  and  data  recording  issues, 
a  significant  effort  is  expended  to  resolve  these  issues.  If  a  program  manager’s 
decisions  demonstrate  that  data  recording  is  a  low  priority,  other  issues  will 
absorb  resources  needed  for  fundamental  data  recording  capability.  However,  if 
the  R2D2  ACB  is  institutionalized,  as  it  has  been  in  the  AEGIS  program,  the 
program  manager  has  the  technical  resources  readily  identified  and  a  chairman 

to  describe  the  status  of  the  data  recording  capabilities  in  the  AEGIS  baseline 
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development.  Additionally,  the  institutionalization  of  the  R2D2  ACB  has  served 
to  educate  members  of  the  program  office  beyond  the  program  manager,  and 
develop  the  expectation  that  a  fundamental  data  recording  capability  is  an 
essential  part  of  the  AEGIS  baseline  development  process  because  the  value 
added  has  been  shown  in  previous  AEGIS  baselines.  Finally,  the  R2D2  ACB 
provides  the  program  manager  a  motivated  group  of  technical  experts  ready  to 
help  resolve  problems  as  they  are  discovered.  Through  the  use  of  technologies 
such  as  video  teleconferencing,  the  R2D2  ACB  has  remained  flexible  and 
responsive  in  support  of  emergent  data  recording  issues  requiring  investigation. 


D.  APPLICATION  OF  STUDY 

1.  Open  Architecture 

The  AEGIS  program  has  been  in  the  process  of  adopting  COTS 
technology  as  quickly  as  possible  without  assuming  too  much  risk  in  the 
program.  Beginning  in  Baseline  6  Phase  1  and  continuing  into  Baseline  7  Phase 
1,  COTS  technology  has  been  inserted.  Baseline  7  Phase  1  is  the  first  AEGIS 
Baseline  to  have  all  COTS  hardware  for  the  AWS.  The  baseline  development 
succeeding  Baseline  7  Phase  1  are  designed  to  be  AEGIS  Open  Architecture 
baselines  with  the  intent  of  capturing  the  complete  benefit  of  COTS. 

The  challenge  facing  the  R2D2  ACB  is  how  to  respond  to  the  changes 
presented  by  open  systems.  The  introduction  of  COTS  in  Baseline  6  Phase  1 
required  the  development  of  the  DXR  data-recording  format.  The  DXR  format 
lacks  features  possessed  by  the  ATES  format  that  are  useful  to  combat  weapon 
system  analysts  due  to  the  significant  differences  between  the  ATES  and  DXR 
formats.  Each  Spiral  represents  a  new  opportunity  to  provide  technical  support 
to  the  AWS  baseline  design  and  development  team.  Information  on  how  to 
improve  the  data  dictionaries  and  data  recording  from  the  previous  Spiral  can  be 
provided.  The  R2D2  ACB  will  provide  a  resource  to  the  design  and  development 
team  as  well  for  assessing  the  impact  of  changes  to  the  AWS  computer  program 
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as  they  are  considered  rather  than  implementing  and  requiring  rework  to  restore 
lost  and  required  capabilities. 

The  issues  that  the  R2D2  ACB  must  resolve  are  critical  to  the  successful 
evaluation  of  the  combat  weapon  system  performance  and  require  a  high  degree 
of  collaboration  throughout  the  entire  AWS  analysis  community.  In  order  for  the 
decisions  of  the  R2D2  ACB  to  be  respected  and  conformed  to,  the  Program 
Manager  must  provide  a  clear  and  unambiguous  level  of  support.  For  AEGIS 
Open  Architecture,  the  real  challenge  is  to  continue  to  add  value  to  the  AEGIS 
program  as  it  is  converted  into  an  open  system. 

2.  DD(X)  Next  Generation  Destroyer 

The  DD(X)  platform  is  being  designed  to  open  architecture  standards.  As 
a  new  program,  most  of  the  people  involved  in  the  DD(X)  program  are  unaware 
of  the  R2D2  ACB.  The  first  step  toward  application  of  the  R2D2  concept  is 
information  and  education.  The  program  managers  in  the  DD(X)  program  are 
unable  to  implement  a  structure  similar  to  the  R2D2  ACB  if  they  are  not  aware  of 
its  benefits.  Additionally,  the  program  managers  need  to  be  introduced  to  the 
people  who  can  provide  the  expertise  to  initiate  this  type  of  ACB.  Once  the 
program  managers  decide  to  implement  a  R2D2  ACB,  the  system  designers 
need  to  be  introduced  to  the  concept  of  the  R2D2  ACB  and  the  systems 
engineers  who  will  be  analyzing  the  performance  of  the  combat  weapon  system 
need  to  be  identified.  With  the  program  managers,  system  designers  and 
analysts  identified,  the  R2D2  ACB  can  start  exchanging  information. 
Traditionally,  the  R2D2  ACB  has  used  the  VTC  format  along  with  an  electronic 
document  management  system.  The  choice  of  mediums  used  for  information 
exchanges  can  be  tailored  to  those  that  will  be  the  most  effective  for  the 
participants.  Direct  meetings  can  be  more  effective  if  all  of  the  participants  are 
closely  located  and  email  can  facilitate  a  very  large  distribution  list  of  participants. 
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Figure  16.  DD(X)  R2D2  ACB  Structure 


With  the  development  of  a  number  of  new  systems,  a  possible  application 
of  the  R2D2  ACB  is  tailored  approach  of  semi-autonomous  groups  that  are 
assigned  to  each  element  or  group  of  elements.  A  possible  structure  for  this 
approach  is  shown  in  Figure  16.  By  using  functional  working  groups,  the 
program  managers  could  direct  resources  to  the  area  or  areas  with  the  most 
critical  problems  that  should  reduce  the  overall  risk  in  the  area  of  data 
dictionaries  and  data  recording.  As  the  program  progresses  through  its 
development  cycle,  the  R2D2  ACB  will  be  in  place  to  support  the  formal  Test  and 
Evaluation  events  such  as  Developmental  and  Operational  Testing  that  have 
relied  heavily  on  recorded  test  data  for  system  evaluation. 
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V.  CONCLUSIONS 


A.  INTRODUCTION 


The  R2D2  ACB  has  been  an  effective  tool  available  to  AEGIS  program 
managers  since  its  inception  in  July  2001.  Initially,  the  focus  of  the  R2D2  ACB 
was  on  issues  related  to  the  ADS  element  of  the  AWS.  As  the  R2D2  ACB 
continued  to  develop,  issues  related  to  each  of  the  AEGIS  core  elements  were 
investigated  and  potential  solutions  were  proposed.  While  the  team  desires  to 
see  every  solution  implemented,  the  Program  Manager  is  the  final  authority  in 
determining  which  solutions  are  to  be  pursued.  The  Program  Manager  must  find 
the  funding  for  issue  resolution  and  is  in  the  best  position  to  evaluate  trade-offs 
between  options.  Based  upon  past  experience  of  the  R2D2  ACB,  the  Program 
Manager  is  not  deciding  between  which  data  dictionary  and  data  recording 
issues  that  should  be  funded.  Rather,  the  decision  is  between  data  dictionary 
and  data  recording  issues  versus  issues  in  other  areas  such  as  software  code 
fixes  or  documentation.  While  the  R2D2  ACB  has  had  some  significant 
accomplishments,  change  will  be  required  if  it  is  to  continue  growing  into  the 
future  and  increase  the  value  that  it  can  add  to  programs. 

B.  KEY  POINTS  AND  RECOMMENDATIONS 


Overall,  the  R2D2  ACB  has  been  successful  in  its  efforts  supporting  the 
AEGIS  program.  Participation  has  included  most  of  the  stakeholders  in  the 
AEGIS  community  who  receive  recorded  data  and  rely  on  that  data  to  perform 
systems  engineering  analysis.  The  key  recommendations  are: 

1.  R2D2  ACB  lacks  direct  funding.  The  R2D2  ACB  lacks  authority 
through  funding  mechanisms  to  prioritize  problems  in  data  dictionaries  and  data 
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recording.  Until  data  dictionary  and  data  recording  is  specifically  written  into  the 
acquisition  contracts  and  direct  funding  is  attached,  it  will  remain  vulnerable  and 
expendable. 

2.  Membership  stability.  When  data  recording  is  not  viewed  as  a 
high  priority,  the  participation  is  inconsistent.  Personnel  turnover  is  inevitable  in 
any  program  but  the  transition  often  does  not  encompass  R2D2  responsibilities. 
The  result  is  a  loss  of  participation  from  some  activities. 

3.  Issues  found  during  or  after  formal  acceptance  testing. 

CSSQT  testing  is  a  critical  event  in  the  delivery  of  the  AWS  for  introduction  to  the 
fleet  or  preparation  for  DT  and/or  OT.  It  has  been  common  to  have  many 
unknowns  in  the  data  recording  capabilities  entering  these  tests.  During  the  USS 
Winston  S.  Churchill  (DDG-81)  CSSQT,  the  analysis  team  noted  24  weapon 
system  performance  analysis  issues  of  which  7  issues  were  related  to  data 
dictionaries  and  data  recording.  With  uncertainties  in  the  capability  of  the  data 
dictionaries  and  data  recording,  necessary  data  could  be  lost  or  not  available 
resulting  in  the  inability  to  characterize  the  system  performed.  A  critical  part  of 
the  CSSQT  testing  is  to  allow  Test  Objectives  to  be  answered  to  characterize 
critical  functionality  with  the  AWS.  This  would  hinder  the  productivity  of  the 
CSSQT  and  possibly  leave  many  Test  Objectives  unanswered  simply  because  of 
data  recording  deficiencies  that  may  have  been  easily  remedied  prior  to  the 
testing. 

4.  Collaborative  Analysis.  The  R2D2  ACB  allows  the  analysis 
community  to  bring  data  dictionary  and  data  recording  issues  to  the  table  that 
may  have  otherwise  gone  undiscovered  during  formal  acceptance  testing  such 
as  system  demonstration  tests.  Working  together,  as  the  R2D2  ACB  identifies 
an  issue,  the  analysis  community  can  understand  and  find  solutions  to  help 
mitigate  or  resolve  the  issue.  The  collaboration  has  proven  to  be  more  effective 
than  having  each  activity  analyze  issues  independently.  With  the  complexity  of 
military  systems,  a  collaborative  approach  will  be  essential  in  the  future  as  it  is 
unlikely  that  a  single  person  or  group  will  be  able  to  evaluate  system 
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performance.  The  R2D2  ACB  provides  the  opportunity  to  bring  together  a 
focused  group  of  experts  capable  of  determining  the  data  dictionary  and  data 
recording  requirements  to  support  the  entire  analysis  community.  Loss  of  the 
R2D2  ACB  would  likely  result  in  a  return  to  inconsistent  analysis  methods  across 
system  elements  and  activities.  It  is  the  membership  that  has  made  the  R2D2 
ACB  strong  and  viable. 

5.  Closed  Loop  Systems  Engineering  Process.  The  R2D2  ACB 
has  been  a  part  of  the  closed  loop  systems  engineering  process  that  the  AEGIS 
program  has  implemented.  While  the  AEGIS  program  has  only  been  able  to 
achieve  a  partial  implementation  of  the  closed  loop  systems  engineering  process, 
R2D2  ACB  can  serve  as  the  catalyst  in  continuing  the  closed  loop  systems 
engineering  process  into  other  combat  weapon  system  platforms  as  the  AEGIS 
program  completes  the  baseline  development  efforts. 

By  introducing  the  R2D2  ACB  into  the  program  at  the  initial  design  stages, 
the  potential  for  significant  reductions  in  data  dictionary  and  data  recording 
rework  efforts  exists.  Data  recording  can  help  to  produce  a  higher  quality 
software  product  by  assisting  the  development  effort.  A  robust  and  effective  data 
recording  capability  allows  software  rework  to  be  objectively  assessed  and 
assists  the  developer  in  determining  if  software  rework  has  caused  inadvertent 
changes  in  the  computer  program  performance.  The  experience  of  the  R2D2 
ACB  in  the  AEGIS  program  has  demonstrated  that  a  significant  number  of 
problems  continue  to  exist  beyond  the  rework  to  the  AEGIS  computer  program 
that  has  already  been  done. 

6.  Early  Introduction  in  System  Development. 

From  the  survey  results,  the  general  consensus  is  that  the  R2D2  ACB  can 
provide  great  benefit  and  be  very  effective  for  a  new  weapon  system  program. 
However,  it  is  critical  to  be  involved  early  when  the  system  is  designed  to  ensure 
that  the  requirements  in  the  system  design  include  the  infrastructure  necessary 
to  capture  the  data  necessary  to  characterize  system  performance.  Data 
recording  is  actually  an  important  part  of  ensuring  that  a  system  will  be 

53 


supportable.  The  information  on  system  performance  that  is  provided  by  data 
recording  and  reduction  can  provide  evidence  of  system  deficiencies  and 
reliability,  maintenance  and  availability.  In  fact,  as  testing  for  events  such  as 
early  operational  assessments  are  integrated  into  program  schedules,  the 
effectiveness  of  the  R2D2  ACB  will  increase.  Early  involvement  applies  to 
existing  systems  that  are  being  upgraded  as  well.  As  the  next  generation  of  data 
recording  is  created  within  the  AEGIS  Open  Architecture  design,  the  R2D2  ACB 
will  need  to  be  involved  in  the  definition  and  implementation  of  the  data  dictionary 
and  data  recording  formats  as  they  are  designed. 


C.  POTENTIAL  AREAS  FOR  FUTURE  RESEARCH 


Many  areas  exist  for  future  research  on  this  topic.  Since  this  study  has 
been  limited  to  combat  weapon  systems  within  the  Navy,  future  research  could 
pursue  the  applicability  of  the  R2D2  ACB  to  weapon  systems  in  other  branches 
of  the  military  service.  Additionally,  programs  that  rely  on  mission  critical 
systems  such  as  the  Air  Traffic  Control  Systems  of  the  Federal  Aviation 
Administration  (FAA)  and  the  Space  Shuttle  Program  of  the  National  Aeronautics 
and  Space  Administration  (NASA)  are  candidates  for  future  research  as  well. 

D.  CLOSING  REMARKS 

The  results  of  the  survey  indicate  that  the  R2D2  ACB  can  provide  value  to 
a  weapon  system  program  but  value  is  not  the  only  criteria  that  a  Program 
Manager  must  consider.  Two  additional  considerations  are  whether  the  R2D2 
ACB  aligns  with  the  Defense  Transformation  efforts  underway  and  will  it  reduce 
risk  to  the  program  in  a  concurrent  development  environment.  For  the  Defense 
Transformation  efforts,  this  analysis  will  focus  on  the  transformation  as  provided 
in  SEA  POWER  21.  As  the  Navy  moves  toward  Network  Centric  Warfare,  the 
data  shared  and  transferred  among  systems  will  become  more  critical  than  in 
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previous  generations  of  combat  weapon  systems.  Another  consideration  is  in  the 
area  of  concurrent  system  development.  Concurrent  system  development 
results  in  early  operational  assessment  of  system  performance.  As  formal 
system  testing  is  conducted  earlier  in  the  system  development,  the  capability  to 
assess  system  performance  is  being  required  earlier  in  the  system  development. 
Data  recording  and  reduction  provide  the  objective  evidence  to  establish  system 
design  maturity  and  document  operational  issues  in  system  performance.  The 
R2D2  ACB  has  demonstrated  that  it  will  provide  value  to  the  Program  Manager 
by  prioritizing  data  recording  and  reduction  capability  requirements  allowing  the 
Program  Manager  to  trade-off  value  added  among  competing  priorities. 
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APPENDIX  A. 


A.  TOR  FORM 


To:  LMSD  SSAAWCS 

c/o  Computer  Sciences  Corporation 
Aegis  Configuration  Management  Office 

P.O.  Box  N,  Moorestown,  NJ  08057 


ORIGINATOR  INFORMATION  (Required) 


3.  Originator  Name 

4.  Organization  □  RAYTHEON 

□  LMSysEng  □  CSC  □  TECHREP 

□  LMCPP  □  NSWC  □  ATT 

□  LMS/WDev.  □  CMU 

5.  System  Element 

6.  Date  Observed 

7.  Return  Address 

8.  Program 

Identification/Loadfile 

9.  Baseline 

7P1 

10. 

11. 

12.  Test  Site  (where  observed)  □  NSWC  □  ACSC  (Wallops  Island) 

□  Desk  Check  □  PGCD  CPTSD  CSEDS  □  PTC 

□  □ 

13.  Test  type  □  MEIT  □  System  Level  Eng.  Tests  (ISEs,  Stress,  Endurance,  etc.)  □  SQT/EQT 

□  Shipyard  Checkout  □  Element  Int.  □  CSIT  □  PTC  Checkout/Production  Acceptance  Test  □  Shipboard 

Operation 

□  Dev/SE  Checkout  □  ET&E  □  Formal  Demos,  DTs/OTs,etc. 

ORIGINATOR  INFORMATION  (As  applicable) 


14.  Test 

Environment 

15.  Test  Procedure  Used 

S  S 

cript  tep 

16.  Dump/Recorded  Data 

TDD 
ape  Nos.:  ump  ata 

D  P 

17.  Documentation  Affected 

18.  Module/Function  Affected 

19.  Site  Log  No. 

19a.  ITT  Cross  Reference 

19b.  Japan  CPCR  Number 

20.  Associated  TDR 

ORIGINATOR  OBSERVATION  (Required) 

21 .  Short  Title  (Reflective  of  Observation) 


22.  Description  of  Observation,  GMT  (if  applicable),  and  other  comments  (Unclassified  Information  only) 


CONFIGURATION  MANAGEMENT  INFO.  Page  1 


1 .  TOR  Log 

2.  TOR  Number 

Date 

T 

_ Attachment  □ 

23.  Effect  on  System  Element  when  observation  described  occurs  and  additional  System  Engineering  comments  (if 
applicable).  (If  recommended  operational  priority  is  3,  describe  workaround.) 


Pri 

Check  one  of  each  or  N/A: 

Sec. 

Recommended  1679A  Operational 

□  1  □  2  □  3  □  4  □  5 

Recommended  Operational 

□  ADBacnDDE 

24.  Local  Problem  Report  (LPR)  Determination 
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a.  QA  Load/CTL?  DYES  □  NO  If  yes,  do  not 

b.  If  no: 

Problem  in  new  development  code?  □  YES  □  NO  □ 

c.  If  b  is  yes,  identify  #  of  CRCR  that 
and  treat  TOR  as  an  LPR 


25. 

CPCR# 

(For 

non-LPRs 

SIB  Pri 

CPCR  Sub 
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APPENDIX  B 


A.  EXAMPLE  PRESENTATION  FOR  R2D2  ACB 


BASELINE  6  PHASE  3 
DATA  ANALYSIS 
LESSONS  LEARNED 


m 

XL/** 

7f^ 

— ^  jri  Jl 

o 

"V  s 

^  / 

O 
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PURPOSE 


Preparing  for  CSSQT 

•  CSEDS  Testing 

•  Analysis  Tools 

•  Dahlgen  Test  Team  Support 

•  Shipyard  and  beyond 


CSSQT  Lessons  Learned 


Event  Example 
Data  Quantities 
Tools 
Timelines 


Have  we  learned  our  lessons? 


PA312 
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Preparing  for  CSSQT 


CSEDS  Testing 

■  In  Baseline  6  Phase  3,  it  became  apparent  very  early  on  that  the 
status  quo  was  not  going  to  work.  Experience  with  Adjunct  data  from 
6  phase  1  drove  us  to  CSEDS. 

■  Functions  at  CSEDS:  Tool  development  and  debugging,  system 
knowledge,  interfacing  with  design  agent,  DEMO  testing  support  in 
DTD  analysis  and  verification.  DEMO  testing  is  more  than  SPY 
analysis. 

■  Analysis  experience  with  CSSQT  Dry  Runs!  Testing  of  tools  with  data 
that  is  as  comparable  to  real  life  as  can  be  achieved  in  a  lab  setting. 

Without  the  knowledge  and  analysis  process  and  tool 
development,  the  completion  of  event  reconstruction 
of  the  6  Phase  3  CSSQTs  would  not  have  been  very  successful 
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Analysis  Tools 

Analysis  Tools  required  more  effort  than  expected 

■  Configuration  Management  has  been  implemented 

•  Challenges  still  continue  in  the  trade-off  between 
responsiveness  versus  control 

■  CSEDS  testing  provided  some  structured  testing  and  in  some 
instances  the  ability  to  redo  some  things. 

•  Trying  to  redo  things  at  CSSQT  is  expensive  and  often  just  can’t 
be  done 

•  Written  procedure  makes  it  much  easier  to  verify  analysis  tool 
performance 

Lack  of  CEC  integration  during  the  CSEDS  tests 

resulted  in  some  unexpected  tool  deficiencies 

FUJ  I  4 
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CSSQT  Example 
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Evaluation  Criteria  For  Surface 
Missile  System  (SMS)  Performance 


NAVSEA  INSTRUCTION  C8820.1 

•  To  provide  criteria  for  assessing  SMS  combat  systems 
performance  during  AAW  non-combat  firing  operations.  This 
criteria  provides  the  measure  for  assessing  performance  against 
air  and  surface  targets,  consistent  with  realistically  observed 
data. 

The  Criteria 

•  Combat  system  performance  during  a  target  presentation 
consists  of  three  phases 

•  Surveillance  Phase 

•  Target  Engagement  Phase 

•  Missile  firing  Phase 
AWS  Test  Objectives 

•  Primary  -  226 

•  Secondary  -  58,  302,  441 
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A23-D8602 


DDG SS Trach  Pont: 
SGO'TQaODMinom  DDG  3G 
S3 '30'  N;  11 9' 45'  ■A1' 


Not  to  exact  Scale 


Surface  VAirfarft  Centw  DMgIoii 


A23-R650: 


A23-D8602 


□  BQM34S,  SSJ  HP  Blink  5fi,  DSQ50 
□ALT:  0 .05 Kft ,  SPD:  M0.7 

□  Heading:  1473  T 
□30  DM  IP,  0  DM  CPA 

□  NKC-135  SOJ  LP  Barrage 

□  Heading:  143  T 

□  175  DM  IP 


, 
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How  It  Will  Be  Scored 


One  Presentation 
One  Target 

Demonstrate  Depth  Of  Fire 

Coverage  Area 
TO-226  Primary 


NKC 


Coverage  Area 
TO-441  Primary 

Complies  With  C8820.1 
Two  Primary  TO’s  Can  Work 
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AN/SPY 

OPTICAL 

Normal  operation  is  for  the  data 
processor  on  the  ship  to  take  the 
optical  and  divide  it  into  its 
partitions,  zip  it  and  send  it  to 
Corona. 

For  15  Partitions  it  takes  45  min  to 
transfer  all  compressed  files.  (3 
minutes  per  partitions) 

Once  received,  the  on  site  data 
processors  do  a  sort  and  split  of 
the  partition  into  its  “ATES”  and 
“DXR”  Components.  For  15 
partitions  this  takes  30  minutes. 
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:Track  Data 
■C&D  Traffic 
iTip  Data 

:  Search  Data 


AN/SPY 

OPTICAL 


Optical,  -  2gig 
Once  received  is 
split  into  up  to 
30  time  aligned 
partitions. 

PA3U0  &2W21 )« 


Piittoul 

PilttonJ 

PiltitiGllS 

PiitiiBii+ 

PiitiiBiij 

PiltitiGllL 

PiittonT 

w 

P4ltiii&nS 

PiltitiGllS 

PiltitiGnlO 

p.ixtii.iLll 

p.ixtii.iLlI 

PiltitiGlllS 

PiitiiBiil+ 

PiitiiBiilj 

Data  is  NOT  time  aligned,  but  recorded 
as  a  function  of  when  a  buffer  is  full,  or 
during  a  preprogrammed  periodic,  it 
will  be  sent  to  the  optical 


ATES 

DXR 

For  Partition 
7,  all  data 
can  be 
extracted 
because  of 
the  luck  in 
buffer 
sequence 
loading 


For  Partition  15,  to  get  all 
the  data  you  have  to  go  back 
to  partition  12,  and  maybe 
into  the  next  optical  for  the 
data 
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What  it  takes 

■  For  every  hour  of  testing  in  manned  raids,  it  takes  4  hours 

•  To  get  the  data  here, 

•  Process  the  data, 

•  Evaluate  the  test  objectives, 

•  Put  together  the  report 

■  CEC  data  is  a  critical  player  in  every  event.  Remote  CEP  data  is 
the  driving  source  of  most  engagements.  Every  live  fire 
engagement  for  the  recent  86/88  CSSQT  had  CEC  data  in  it 

■  DXR  data  is  cumbersome  to  go  through.  Measured  in  GB  instead 
of  MB! 

■  Limitations  still  exist 

•  We  have  no  clues  as  to  what  happening  on  the  LAN 

•  Adjunct  switching  and  reconfiguration  is  not  recorded 

Since  DDG-85,  data  prep  time  required  for  event  analysis 
has  doubled  and  in  SPY  and  C&D  tripled. 

PA3111 
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Lessons  Learned? 


■  We  will  be  doing  BASELINE-7 

■  All  COTS . 

■  Optical’s  are  5  GB,  twice  the  capacity  as  6  Ph  III 

■  New  Programs 

■  All  DXR  data 

■  All  new  Hardware  with  increased  performance  capability.  The 
big  Littoral  radar  system 

■  Anticipate  more  analysis  time  than  with  the  6  Ph  III  ships 

The  key  to  success  is  early,  sustained  effort 
wherever  the  opportunities  exist 
Teaming  with  TECHREP,  NGIT,  Dahlgren,  LM 
and  all  other  stakeholders 

PA3U2 
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